2020-09-08T16:27:22.961 INFO:root:teuthology version: 1.0.0-55bd2dc 2020-09-08T16:27:22.964 DEBUG:teuthology.run:Teuthology command: teuthology --lock --archive /home/teuthworker/archive/kchai-2020-09-08_12:29:35-rados-wip-kefu-testing-2020-09-08-0934-distro-basic-smithi/5417978 --name kchai-2020-09-08_12:29:35-rados-wip-kefu-testing-2020-09-08-0934-distro-basic-smithi --block --owner scheduled_kchai@teuthology --description rados/standalone/{supported-random-distro$/{centos_8} workloads/erasure-code} --verbose -- /tmp/teuthology-worker.ig09px4k.tmp 2020-09-08T16:27:23.293 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-09-08T16:27:23.367 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2020-09-08_12:29:35-rados-wip-kefu-testing-2020-09-08-0934-distro-basic-smithi/5417978 branch: wip-kefu-testing-2020-09-08-0934 description: rados/standalone/{supported-random-distro$/{centos_8} workloads/erasure-code} email: tchaikov@gmail.com first_in_suite: false job_id: '5417978' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2020-09-08_12:29:35-rados-wip-kefu-testing-2020-09-08-0934-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-kefu-testing-2020-09-08-0934 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: dce4d291bd4243c9802c17d67b85a6da6bbc0f8f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: dce4d291bd4243c9802c17d67b85a6da6bbc0f8f selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-kefu-testing-2020-09-08-0934 sha1: dce4d291bd4243c9802c17d67b85a6da6bbc0f8f owner: scheduled_kchai@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: dce4d291bd4243c9802c17d67b85a6da6bbc0f8f sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2020-09-08-0934 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-09-08-0934/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: dce4d291bd4243c9802c17d67b85a6da6bbc0f8f tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology_branch: master timestamp: 2020-09-08_12:29:35 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.12104 2020-09-08T16:27:23.367 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-09-08-0934/qa; will attempt to use it 2020-09-08T16:27:23.368 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-09-08-0934/qa/tasks 2020-09-08T16:27:23.368 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-09-08T16:27:23.370 INFO:teuthology.task.internal:Checking packages... 2020-09-08T16:27:23.401 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'dce4d291bd4243c9802c17d67b85a6da6bbc0f8f' 2020-09-08T16:27:23.402 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-09-08T16:27:23.403 INFO:teuthology.packaging:ref: None 2020-09-08T16:27:23.403 INFO:teuthology.packaging:tag: None 2020-09-08T16:27:23.403 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-09-08-0934 2020-09-08T16:27:23.404 INFO:teuthology.packaging:sha1: dce4d291bd4243c9802c17d67b85a6da6bbc0f8f 2020-09-08T16:27:23.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2020-09-08-0934 2020-09-08T16:27:23.581 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-5272.gdce4d291bd4 2020-09-08T16:27:23.582 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-09-08T16:27:23.594 INFO:teuthology.task.internal:no buildpackages task found 2020-09-08T16:27:23.595 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-09-08T16:27:23.648 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-09-08T16:27:23.649 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-09-08T16:27:23.702 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 1, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-09-08_12:29:35-rados-wip-kefu-testing-2020-09-08-0934-distro-basic-smithi/5417978'} 2020-09-08T16:27:23.729 DEBUG:teuthology.lock.ops:locked ubuntu@smithi120.front.sepia.ceph.com 2020-09-08T16:27:23.768 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7cf3d372f0>, signals=[15]) 2020-09-08T16:27:23.984 INFO:teuthology.provision.fog.smithi120:Scheduling deploy of centos 8.1 2020-09-08T16:27:24.345 INFO:teuthology.orchestra.console:Power off smithi120 2020-09-08T16:27:24.347 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-09-08T16:27:24.387 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T16:27:28.426 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T16:27:32.566 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T16:27:32.787 INFO:teuthology.orchestra.console:Power off for smithi120 completed 2020-09-08T16:27:32.889 INFO:teuthology.orchestra.console:Power on smithi120 2020-09-08T16:27:32.890 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-09-08T16:27:32.925 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T16:27:36.962 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T16:27:41.079 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-09-08T16:27:41.298 INFO:teuthology.orchestra.console:Power on for smithi120 completed 2020-09-08T16:27:41.402 INFO:teuthology.provision.fog.smithi120:Waiting for deploy to finish 2020-09-08T16:30:12.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:30:22.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:30:31.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:30:40.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:30:49.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:30:58.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:31:07.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:32:13.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:32:13.575 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-09-08T16:32:13.579 INFO:teuthology.orchestra.run.smithi120:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-09-08T16:32:13.913 INFO:teuthology.provision.fog.smithi120:Node is ready 2020-09-08T16:32:13.914 INFO:teuthology.orchestra.run.smithi120:> hostname 2020-09-08T16:32:13.935 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120.front.sepia.ceph.com 2020-09-08T16:32:13.935 INFO:teuthology.orchestra.run.smithi120:> grep smithi120.front.sepia.ceph.com /etc/hosts 2020-09-08T16:32:13.989 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.15.120 smithi120.front.sepia.ceph.com smithi120 2020-09-08T16:32:13.990 INFO:teuthology.orchestra.run.smithi120:> sudo hostname smithi120 2020-09-08T16:32:14.111 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i -e 's/smithi120.front.sepia.ceph.com/smithi120/g' /etc/hosts 2020-09-08T16:32:14.138 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i -e 's/172.21.15.120/172.21.15.120/g' /etc/hosts 2020-09-08T16:32:14.205 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i -e 's/smithi120.front.sepia.ceph.com/smithi120/g' /etc/hostname 2020-09-08T16:32:14.272 INFO:teuthology.orchestra.run.smithi120:> sudo hostname smithi120 2020-09-08T16:32:14.338 INFO:teuthology.provision.fog.smithi120:Deploy complete! 2020-09-08T16:32:14.340 DEBUG:teuthology.parallel:result is None 2020-09-08T16:32:14.440 INFO:teuthology.lock.ops:Checking smithi120.front.sepia.ceph.com 2020-09-08T16:32:14.442 INFO:teuthology.lock.ops:New key found. Updating... 2020-09-08T16:32:14.461 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: set os type and version on server 2020-09-08T16:32:14.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-09-08T16:32:14.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:32:14.533 INFO:teuthology.orchestra.run.smithi120:> true 2020-09-08T16:32:14.687 INFO:teuthology.orchestra.run.smithi120:> uname -m 2020-09-08T16:32:14.740 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2020-09-08T16:32:14.740 INFO:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2020-09-08T16:32:14.793 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Linux" 2020-09-08T16:32:14.793 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8 (Core)" 2020-09-08T16:32:14.794 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2020-09-08T16:32:14.794 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2020-09-08T16:32:14.794 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8" 2020-09-08T16:32:14.794 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8" 2020-09-08T16:32:14.795 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-09-08T16:32:14.795 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2020-09-08T16:32:14.795 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-09-08T16:32:14.795 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.centos.org/" 2020-09-08T16:32:14.795 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-09-08T16:32:14.796 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:32:14.796 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-09-08T16:32:14.796 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-09-08T16:32:14.796 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-09-08T16:32:14.796 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-09-08T16:32:14.797 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:32:14.798 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2020-09-08T16:32:14.815 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-09-08T16:32:14.831 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjpIizRGoQCDHyHjIjE2B1QhxxQdMXQ51ssu0KtvliOq0+ayX5pj8EhuzVK+FDjVB2cpD2Om8MZczTJdaCIIYNq6ZY6ljwGuRnhNs7hvvuLM/E+gt/SJ93RI1U5djpXV5bDlmUy5OQJEtLc4rVolOUhvHDs3TOCLWQIaK+23Q5862cOB2QF3Xu2Stryk2mXWWiW+zvcqwOEPtLIrCMS4I13iId0hcQFj/otxygmYwJRddenH2URflpzEn80XH8QsyBe0M+cmqN1DTx30kkDBCDzQn4YrRVT1Wz+FRFLNTQav8G8hYwx3eSPzDrux8ZAb4l2X3u4GYbw0hPYECrvDoDDjijFVCKrxDbT+31L1dp2TbwkEjnVAaImtSq5NXppUTsVi3Nal2cbsrDIUYeqBehiy/On2PvocVNGuszXv48YaOP/38ZUHIBRZ0Igv7+Rs4psOQ9Ayv+tnqChilhtyDxZ60zdbvVz/C4uA18/h6Qr8LWqudw1EoFd4uUVPqaQ0U= 2020-09-08T16:32:14.832 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-09-08T16:32:14.875 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-09-08T16:32:14.885 INFO:teuthology.task.internal:Saving configuration 2020-09-08T16:32:14.895 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-09-08T16:32:42.487 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-09-08T16:32:42.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-09-08 16:27:23.718403', 'description': '/home/teuthworker/archive/kchai-2020-09-08_12:29:35-rados-wip-kefu-testing-2020-09-08-0934-distro-basic-smithi/5417978', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDjhU3gh6aHgdLblarvS9ojiWM37JBhFxUWl67mxXYjD3BtKR9ZTvo0B/JbMo7cc1LLUzOBzLC0slSJ3+L6jGwE=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi120.front.sepia.ceph.com'} 2020-09-08T16:32:42.509 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-09-08T16:32:42.516 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-09-08T16:32:42.517 INFO:teuthology.run_tasks:Running task console_log... 2020-09-08T16:32:42.557 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7cf6827268>, signals=[15]) 2020-09-08T16:32:42.565 INFO:teuthology.run_tasks:Running task internal.connect... 2020-09-08T16:32:42.575 INFO:teuthology.task.internal:Opening connections... 2020-09-08T16:32:42.575 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2020-09-08T16:32:42.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:32:42.651 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-09-08T16:32:42.658 INFO:teuthology.orchestra.run.smithi120:> uname -m 2020-09-08T16:32:42.817 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2020-09-08T16:32:42.818 INFO:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2020-09-08T16:32:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Linux" 2020-09-08T16:32:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8 (Core)" 2020-09-08T16:32:42.871 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2020-09-08T16:32:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2020-09-08T16:32:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8" 2020-09-08T16:32:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8" 2020-09-08T16:32:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-09-08T16:32:42.872 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2020-09-08T16:32:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-09-08T16:32:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.centos.org/" 2020-09-08T16:32:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-09-08T16:32:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:32:42.873 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-09-08T16:32:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-09-08T16:32:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-09-08T16:32:42.874 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-09-08T16:32:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:32:42.875 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2020-09-08T16:32:42.891 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-09-08T16:35:18.991 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-09-08T16:35:19.007 INFO:teuthology.task.internal:Checking for old test directory... 2020-09-08T16:35:19.009 INFO:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2020-09-08T16:35:19.035 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-09-08T16:35:19.045 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-09-08T16:35:19.046 INFO:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2020-09-08T16:35:19.106 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-09-08T16:35:19.144 INFO:teuthology.run_tasks:Running task kernel... 2020-09-08T16:35:19.165 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2020-09-08T16:35:19.166 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-09-08T16:35:19.166 INFO:teuthology.orchestra.run.smithi120:> uname -r 2020-09-08T16:35:19.185 INFO:teuthology.orchestra.run.smithi120.stdout:4.18.0-193.14.2.el8_2.x86_64 2020-09-08T16:35:19.185 INFO:teuthology.task.kernel:Running kernel on smithi120: 4.18.0-193.14.2.el8_2.x86_64 2020-09-08T16:35:19.186 INFO:teuthology.orchestra.run.smithi120:> sudo yum install -y kernel 2020-09-08T16:35:21.798 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - AppStream 39 MB/s | 7.0 MB 00:00 2020-09-08T16:35:22.985 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Base 24 MB/s | 2.2 MB 00:00 2020-09-08T16:35:23.412 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Extras 85 kB/s | 5.5 kB 00:00 2020-09-08T16:35:23.818 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 22 MB/s | 8.0 MB 00:00 2020-09-08T16:35:25.477 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 1.0 MB/s | 21 kB 00:00 2020-09-08T16:35:26.466 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-09-08T16:35:26.466 INFO:teuthology.orchestra.run.smithi120.stdout:Package kernel-4.18.0-193.14.2.el8_2.x86_64 is already installed. 2020-09-08T16:35:26.621 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:35:26.622 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:35:26.622 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:35:26.739 INFO:teuthology.orchestra.run.smithi120:> echo no | sudo yum reinstall kernel || true 2020-09-08T16:35:27.143 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:02 ago on Tue 08 Sep 2020 04:35:25 PM UTC. 2020-09-08T16:35:27.388 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-09-08T16:35:27.388 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-193.14.2.el8_2.x86_64 (from BaseOS) not available. 2020-09-08T16:35:27.436 INFO:teuthology.orchestra.run.smithi120.stderr:Error: No packages marked for reinstall. 2020-09-08T16:35:27.488 INFO:teuthology.orchestra.run.smithi120:> sudo yum reinstall -y kernel || true 2020-09-08T16:35:27.887 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:00:02 ago on Tue 08 Sep 2020 04:35:25 PM UTC. 2020-09-08T16:35:28.154 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-147.el8.x86_64 (from anaconda) not available. 2020-09-08T16:35:28.154 INFO:teuthology.orchestra.run.smithi120.stdout:Installed package kernel-4.18.0-193.14.2.el8_2.x86_64 (from BaseOS) not available. 2020-09-08T16:35:28.205 INFO:teuthology.orchestra.run.smithi120.stderr:Error: No packages marked for reinstall. 2020-09-08T16:35:28.252 INFO:teuthology.orchestra.run.smithi120:> rpm -q kernel --last | head -n 1 2020-09-08T16:35:28.310 INFO:teuthology.orchestra.run.smithi120.stdout:kernel-4.18.0-193.14.2.el8_2.x86_64 Tue 01 Sep 2020 04:40:34 PM UTC 2020-09-08T16:35:28.312 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-193.14.2.el8_2.x86_64 2020-09-08T16:35:28.312 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-09-08T16:35:28.312 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-09-08T16:35:28.313 INFO:teuthology.orchestra.run.smithi120:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-09-08T16:35:28.357 INFO:teuthology.orchestra.run.smithi120.stdout:ttyS1 2020-09-08T16:35:28.372 INFO:teuthology.run_tasks:Running task internal.base... 2020-09-08T16:35:28.381 INFO:teuthology.task.internal:Creating test directory... 2020-09-08T16:35:28.381 INFO:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-09-08T16:35:28.395 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-09-08T16:35:28.403 INFO:teuthology.run_tasks:Running task internal.archive... 2020-09-08T16:35:28.416 INFO:teuthology.task.internal:Creating archive directory... 2020-09-08T16:35:28.417 INFO:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-09-08T16:35:28.460 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-09-08T16:35:28.468 INFO:teuthology.task.internal:Enabling coredump saving... 2020-09-08T16:35:28.468 INFO:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-09-08T16:35:28.532 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:35:28.545 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:35:28.546 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-09-08T16:35:28.554 INFO:teuthology.task.internal:Configuring sudo... 2020-09-08T16:35:28.555 INFO:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-09-08T16:35:28.614 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-09-08T16:35:28.624 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-09-08T16:35:28.625 INFO:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-09-08T16:35:28.668 INFO:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-09-08T16:35:28.743 INFO:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-09-08T16:35:28.799 INFO:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-09-08T16:35:28.872 INFO:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-09-08T16:35:28.922 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-09-08T16:35:28.923 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-09-08T16:35:28.975 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-09-08T16:35:28.987 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-09-08T16:35:28.988 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-09-08T16:35:28.988 INFO:teuthology.orchestra.run.smithi120.stderr:152 bytes copied, 2.8076e-05 s, 5.4 MB/s 2020-09-08T16:35:28.989 INFO:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2020-09-08T16:35:29.119 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-09-08T16:35:29.366 INFO:teuthology.run_tasks:Running task internal.timer... 2020-09-08T16:35:29.375 INFO:teuthology.task.internal:Starting timer... 2020-09-08T16:35:29.376 INFO:teuthology.run_tasks:Running task pcp... 2020-09-08T16:35:29.422 INFO:teuthology.run_tasks:Running task selinux... 2020-09-08T16:35:29.450 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-09-08T16:35:29.451 INFO:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2020-09-08T16:35:29.513 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: [ OK ] 2020-09-08T16:35:29.515 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-09-08T16:35:29.515 INFO:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2020-09-08T16:35:29.536 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2020-09-08T16:35:29.536 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2020-09-08T16:35:29.537 INFO:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-09-08T16:35:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-09-08T16:35:29.603 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-09-08T16:35:29.604 INFO:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2020-09-08T16:35:29.685 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-09-08T16:35:29.698 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-09-08T16:35:29.708 INFO:teuthology.repo_utils:Fetching master from origin 2020-09-08T16:35:29.763 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-09-08T16:35:29.779 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-09-08T16:35:29.782 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-09-08T16:35:30.197 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-09-08T16:35:31.251 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-09-08T16:35:33.637 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-09-08T16:35:33.660 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:33 +0000 (0:00:00.078) 0:00:00.078 ***** 2020-09-08T16:35:36.245 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi120.front.sepia.ceph.com] 2020-09-08T16:35:36.326 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:36 +0000 (0:00:02.665) 0:00:02.744 ***** 2020-09-08T16:35:36.813 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-09-08T16:35:36.816 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-09-08T16:35:36.847 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:36 +0000 (0:00:00.521) 0:00:03.265 ***** 2020-09-08T16:35:37.409 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-09-08T16:35:37.411 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-09-08T16:35:37.443 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:37 +0000 (0:00:00.596) 0:00:03.861 ***** 2020-09-08T16:35:37.472 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-09-08T16:35:37.474 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:37.507 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:37 +0000 (0:00:00.062) 0:00:03.924 ***** 2020-09-08T16:35:37.977 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-09-08T16:35:38.012 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:38 +0000 (0:00:00.505) 0:00:04.429 ***** 2020-09-08T16:35:39.049 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-09-08T16:35:39.051 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1599582938.0287442-123190568863872/source", "state": "file", "uid": 0} 2020-09-08T16:35:39.085 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:39 +0000 (0:00:01.073) 0:00:05.503 ***** 2020-09-08T16:35:39.630 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-09-08T16:35:39.632 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-09-08T16:35:39.634 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-09-08T16:35:39.808 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-09-08T16:35:39.815 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-09-08T16:35:39.950 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:39 +0000 (0:00:00.864) 0:00:06.368 ***** 2020-09-08T16:35:40.594 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi120.front.sepia.ceph.com] 2020-09-08T16:35:40.669 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-09-08T16:35:40.670 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:40 +0000 (0:00:00.718) 0:00:07.087 ***** 2020-09-08T16:35:41.294 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-09-08T16:35:41.296 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.245748", "end": "2020-09-08 16:35:41.264014", "rc": 0, "start": "2020-09-08 16:35:41.018266", "stderr": "", "stderr_lines": [], "stdout": "d7c0190f3c79f0107823a9b3caef24ede18680e0", "stdout_lines": ["d7c0190f3c79f0107823a9b3caef24ede18680e0"]} 2020-09-08T16:35:41.337 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.667) 0:00:07.755 ***** 2020-09-08T16:35:41.558 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008739", "end": "2020-09-08 16:35:41.541704", "failed_when_result": false, "rc": 0, "start": "2020-09-08 16:35:41.532965", "stderr": "", "stderr_lines": [], "stdout": "d7c0190f3c79f0107823a9b3caef24ede18680e0", "stdout_lines": ["d7c0190f3c79f0107823a9b3caef24ede18680e0"]} 2020-09-08T16:35:41.592 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.255) 0:00:08.010 ***** 2020-09-08T16:35:41.625 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-09-08T16:35:41.626 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-09-08T16:35:41.662 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.067) 0:00:08.080 ***** 2020-09-08T16:35:41.691 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-09-08T16:35:41.693 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:41.733 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.071) 0:00:08.151 ***** 2020-09-08T16:35:41.763 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-09-08T16:35:41.765 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:41.797 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.064) 0:00:08.215 ***** 2020-09-08T16:35:41.827 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-09-08T16:35:41.829 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:41.861 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.063) 0:00:08.279 ***** 2020-09-08T16:35:41.891 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-09-08T16:35:41.893 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:41.927 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.065) 0:00:08.345 ***** 2020-09-08T16:35:41.956 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-09-08T16:35:41.958 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:41.996 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:41 +0000 (0:00:00.069) 0:00:08.414 ***** 2020-09-08T16:35:42.057 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-09-08T16:35:42.059 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.068 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.072 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.082 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.084 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.094 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.095 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.106 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.108 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.117 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.128 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.130 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.140 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.142 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.152 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.154 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.166 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.174 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.176 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.185 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.187 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.197 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.198 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.208 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.210 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.219 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.221 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.230 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.232 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.242 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.244 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.253 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.255 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.264 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.266 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.276 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.278 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.288 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.290 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.299 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.301 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.311 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.312 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.322 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.324 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.333 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.334 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.334 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.345 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.347 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.357 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.358 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.371 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.373 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.383 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.385 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.394 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.396 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.405 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.416 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.418 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.427 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.431 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.442 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.450 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.470 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.472 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.474 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.484 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.485 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.495 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.497 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.506 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.508 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.518 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.519 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.529 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.530 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.543 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.552 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.554 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.563 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.564 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.574 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.576 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.585 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.587 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.596 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.598 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.611 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.613 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.623 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.625 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.634 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.635 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.647 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.657 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.658 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.683 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.703 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.705 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.716 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.717 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.727 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.728 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.738 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.739 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.749 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.750 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.760 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.761 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.810 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:42 +0000 (0:00:00.813) 0:00:09.228 ***** 2020-09-08T16:35:42.839 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-09-08T16:35:42.874 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:42 +0000 (0:00:00.064) 0:00:09.292 ***** 2020-09-08T16:35:42.913 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-09-08T16:35:42.915 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:42.952 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:42 +0000 (0:00:00.078) 0:00:09.370 ***** 2020-09-08T16:35:42.982 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.017 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:43 +0000 (0:00:00.065) 0:00:09.435 ***** 2020-09-08T16:35:43.048 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-09-08T16:35:43.050 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.084 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:43 +0000 (0:00:00.066) 0:00:09.502 ***** 2020-09-08T16:35:43.116 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-09-08T16:35:43.118 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.153 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:43 +0000 (0:00:00.069) 0:00:09.571 ***** 2020-09-08T16:35:43.183 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-09-08T16:35:43.185 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.219 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:43 +0000 (0:00:00.065) 0:00:09.636 ***** 2020-09-08T16:35:43.259 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-09-08T16:35:43.260 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.278 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.281 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.294 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.305 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.316 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.327 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.329 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.338 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.339 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.349 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.351 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.361 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.363 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.372 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.374 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.383 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.385 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.395 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.396 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.406 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.417 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.424 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.428 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.430 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.460 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.462 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.473 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.475 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:43.521 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:43 +0000 (0:00:00.299) 0:00:09.936 ***** 2020-09-08T16:35:44.178 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-09-08T16:35:44.180 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "e4960b14bdbab8632f61f7e37ba0ea7d7809d19f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-09-08T16:35:44.224 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:44 +0000 (0:00:00.706) 0:00:10.642 ***** 2020-09-08T16:35:44.258 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi120.front.sepia.ceph.com] => { "msg": "Host smithi120.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-09-08T16:35:44.299 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:44 +0000 (0:00:00.074) 0:00:10.717 ***** 2020-09-08T16:35:44.362 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-09-08T16:35:44.364 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-09-08T16:35:44.420 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:44 +0000 (0:00:00.121) 0:00:10.838 ***** 2020-09-08T16:35:44.450 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:44.485 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:44 +0000 (0:00:00.064) 0:00:10.903 ***** 2020-09-08T16:35:45.047 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-09-08T16:35:45.048 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.346091", "end": "2020-09-08 16:35:45.030117", "rc": 0, "start": "2020-09-08 16:35:44.684026", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-09-08T16:35:45.081 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.595) 0:00:11.499 ***** 2020-09-08T16:35:45.121 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-09-08T16:35:45.123 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.156 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.075) 0:00:11.574 ***** 2020-09-08T16:35:45.197 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.235 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.078) 0:00:11.653 ***** 2020-09-08T16:35:45.275 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.310 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.074) 0:00:11.728 ***** 2020-09-08T16:35:45.352 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-09-08T16:35:45.353 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.389 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.079) 0:00:11.807 ***** 2020-09-08T16:35:45.421 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-09-08T16:35:45.422 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.459 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.069) 0:00:11.877 ***** 2020-09-08T16:35:45.491 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-09-08T16:35:45.492 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.529 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.069) 0:00:11.947 ***** 2020-09-08T16:35:45.581 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-09-08T16:35:45.583 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-09-08T16:35:45.622 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.092) 0:00:12.040 ***** 2020-09-08T16:35:45.662 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-09-08T16:35:45.665 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.701 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.079) 0:00:12.119 ***** 2020-09-08T16:35:45.743 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-09-08T16:35:45.745 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.779 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.078) 0:00:12.197 ***** 2020-09-08T16:35:45.824 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-09-08T16:35:45.826 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-09-08T16:35:45.861 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.081) 0:00:12.279 ***** 2020-09-08T16:35:45.902 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-09-08T16:35:45.904 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:45.937 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:45 +0000 (0:00:00.076) 0:00:12.355 ***** 2020-09-08T16:35:45.983 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-09-08T16:35:46.019 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.081) 0:00:12.437 ***** 2020-09-08T16:35:46.060 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-09-08T16:35:46.061 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.097 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.078) 0:00:12.515 ***** 2020-09-08T16:35:46.139 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-09-08T16:35:46.141 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.176 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.078) 0:00:12.594 ***** 2020-09-08T16:35:46.207 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-09-08T16:35:46.209 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.245 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.068) 0:00:12.662 ***** 2020-09-08T16:35:46.286 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-09-08T16:35:46.289 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.322 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.077) 0:00:12.740 ***** 2020-09-08T16:35:46.363 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-09-08T16:35:46.365 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-09-08T16:35:46.400 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.077) 0:00:12.818 ***** 2020-09-08T16:35:46.440 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-09-08T16:35:46.442 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.476 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.076) 0:00:12.894 ***** 2020-09-08T16:35:46.525 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-09-08T16:35:46.527 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.561 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.085) 0:00:12.979 ***** 2020-09-08T16:35:46.613 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-09-08T16:35:46.615 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.650 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.088) 0:00:13.068 ***** 2020-09-08T16:35:46.704 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-09-08T16:35:46.706 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.747 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.097) 0:00:13.165 ***** 2020-09-08T16:35:46.789 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-09-08T16:35:46.791 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.836 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.089) 0:00:13.254 ***** 2020-09-08T16:35:46.876 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-09-08T16:35:46.878 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:46.910 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.073) 0:00:13.328 ***** 2020-09-08T16:35:46.951 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-09-08T16:35:46.953 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-09-08T16:35:46.986 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:46 +0000 (0:00:00.076) 0:00:13.404 ***** 2020-09-08T16:35:47.016 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-09-08T16:35:47.018 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:47.051 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:47 +0000 (0:00:00.064) 0:00:13.469 ***** 2020-09-08T16:35:47.091 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-09-08T16:35:47.093 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:47.125 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:47 +0000 (0:00:00.074) 0:00:13.543 ***** 2020-09-08T16:35:47.364 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-09-08T16:35:47.366 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-09-08T16:35:47.399 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:47 +0000 (0:00:00.273) 0:00:13.817 ***** 2020-09-08T16:35:47.920 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-09-08T16:35:47.922 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-09-08T16:35:48.428 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-09-08T16:35:48.472 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:48 +0000 (0:00:01.072) 0:00:14.890 ***** 2020-09-08T16:35:48.984 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-09-08T16:35:48.986 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-09-08T16:35:49.490 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-09-08T16:35:49.527 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:49 +0000 (0:00:01.055) 0:00:15.945 ***** 2020-09-08T16:35:49.568 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:49.602 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:49 +0000 (0:00:00.074) 0:00:16.020 ***** 2020-09-08T16:35:49.634 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-09-08T16:35:49.635 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:49.673 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:49 +0000 (0:00:00.071) 0:00:16.091 ***** 2020-09-08T16:35:49.675 INFO:teuthology.task.ansible.out: 2020-09-08T16:35:49.705 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-09-08T16:35:49.707 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:49.739 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:49 +0000 (0:00:00.066) 0:00:16.157 ***** 2020-09-08T16:35:49.771 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-09-08T16:35:49.772 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:49.806 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:49 +0000 (0:00:00.066) 0:00:16.224 ***** 2020-09-08T16:35:49.847 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-09-08T16:35:49.848 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:49.884 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:49 +0000 (0:00:00.077) 0:00:16.302 ***** 2020-09-08T16:35:49.914 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-09-08T16:35:49.917 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:49.950 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:49 +0000 (0:00:00.066) 0:00:16.368 ***** 2020-09-08T16:35:49.998 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:50.032 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:50 +0000 (0:00:00.082) 0:00:16.450 ***** 2020-09-08T16:35:50.082 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-09-08T16:35:50.084 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:50.135 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:50 +0000 (0:00:00.102) 0:00:16.553 ***** 2020-09-08T16:35:50.168 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-09-08T16:35:50.171 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:50.204 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:50 +0000 (0:00:00.069) 0:00:16.622 ***** 2020-09-08T16:35:50.515 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-09-08T16:35:50.517 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.102340", "end": "2020-09-08 16:35:50.500350", "rc": 0, "start": "2020-09-08 16:35:50.398010", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:35:50.555 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:50 +0000 (0:00:00.347) 0:00:16.970 ***** 2020-09-08T16:35:51.836 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-09-08T16:35:51.838 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-09-08T16:35:51.878 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:51 +0000 (0:00:01.326) 0:00:18.296 ***** 2020-09-08T16:35:51.922 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-09-08T16:35:51.924 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:51.959 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:51 +0000 (0:00:00.080) 0:00:18.377 ***** 2020-09-08T16:35:51.992 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-09-08T16:35:51.994 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:52.033 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:52 +0000 (0:00:00.073) 0:00:18.451 ***** 2020-09-08T16:35:52.076 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:52.114 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:52 +0000 (0:00:00.081) 0:00:18.532 ***** 2020-09-08T16:35:52.155 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-09-08T16:35:52.157 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:52.189 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:52 +0000 (0:00:00.075) 0:00:18.607 ***** 2020-09-08T16:35:52.719 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-09-08T16:35:52.720 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-09-08T16:35:52.753 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:52 +0000 (0:00:00.563) 0:00:19.171 ***** 2020-09-08T16:35:53.691 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-09-08T16:35:53.732 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:53 +0000 (0:00:00.979) 0:00:20.150 ***** 2020-09-08T16:35:54.255 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-09-08T16:35:54.257 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-09-08T16:35:54.758 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-09-08T16:35:55.226 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-09-08T16:35:55.261 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:55 +0000 (0:00:01.529) 0:00:21.679 ***** 2020-09-08T16:35:55.467 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-09-08T16:35:55.469 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-09-08T16:35:55.519 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:55 +0000 (0:00:00.257) 0:00:21.937 ***** 2020-09-08T16:35:56.023 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-09-08T16:35:56.026 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-09-08T16:35:56.511 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-09-08T16:35:56.990 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-09-08T16:35:57.030 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:57 +0000 (0:00:01.510) 0:00:23.447 ***** 2020-09-08T16:35:57.080 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-09-08T16:35:57.082 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:57.125 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:57 +0000 (0:00:00.095) 0:00:23.543 ***** 2020-09-08T16:35:57.179 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-09-08T16:35:57.181 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-09-08T16:35:57.216 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:57 +0000 (0:00:00.091) 0:00:23.634 ***** 2020-09-08T16:35:57.431 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-09-08T16:35:57.433 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-09-08T16:35:57.471 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:57 +0000 (0:00:00.254) 0:00:23.889 ***** 2020-09-08T16:35:57.518 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-09-08T16:35:57.520 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:57.530 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:57.543 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:57.545 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:57.589 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:57 +0000 (0:00:00.117) 0:00:24.007 ***** 2020-09-08T16:35:58.543 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-09-08T16:35:58.545 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-09-08T16:35:58.578 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:58 +0000 (0:00:00.989) 0:00:24.996 ***** 2020-09-08T16:35:58.609 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-09-08T16:35:58.611 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:35:58.643 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:58 +0000 (0:00:00.064) 0:00:25.061 ***** 2020-09-08T16:35:59.151 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-09-08T16:35:59.153 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-09-08T16:35:59.188 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:59 +0000 (0:00:00.544) 0:00:25.606 ***** 2020-09-08T16:35:59.712 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-09-08T16:35:59.714 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-09-08T16:35:59.748 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:59 +0000 (0:00:00.560) 0:00:26.166 ***** 2020-09-08T16:35:59.792 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-09-08T16:35:59.794 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:59.803 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-09-08T16:35:59.842 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:35:59 +0000 (0:00:00.093) 0:00:26.259 ***** 2020-09-08T16:36:00.073 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-09-08T16:36:00.108 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:00 +0000 (0:00:00.266) 0:00:26.526 ***** 2020-09-08T16:36:00.380 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-09-08T16:36:00.383 INFO:teuthology.task.ansible.out:fatal: [smithi120.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-09-08T16:36:00.422 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:00 +0000 (0:00:00.310) 0:00:26.836 ***** 2020-09-08T16:36:00.461 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-09-08T16:36:00.463 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:00.496 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:00 +0000 (0:00:00.078) 0:00:26.914 ***** 2020-09-08T16:36:01.036 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-09-08T16:36:01.038 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2020-09-08T16:36:01.081 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:01 +0000 (0:00:00.584) 0:00:27.499 ***** 2020-09-08T16:36:01.942 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-09-08T16:36:01.946 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2020-09-08 16:31:35 UTC", "ActiveEnterTimestampMonotonic": "16071774", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount time-sync.target network.target systemd-journald.socket basic.target -.mount local-fs.target var-run.mount nss-lookup.target sysinit.target system.slice systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2020-09-08 16:31:35 UTC", "AssertTimestampMonotonic": "16070997", "Before": "shutdown.target multi-user.target plymouth-quit.service getty@tty1.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2020-09-08 16:31:35 UTC", "ConditionTimestampMonotonic": "16070997", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-09-08T16:36:01.948 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2500", "ExecMainStartTimestamp": "Tue 2020-09-08 16:31:35 UTC", "ExecMainStartTimestampMonotonic": "16071741", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Tue 2020-09-08 16:31:35 UTC] ; stop_time=[n/a] ; pid=2500 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2020-09-08 16:31:35 UTC", "InactiveExitTimestampMonotonic": "16071774", "InvocationID": "a730aab4d3934909b7ffffc0659c494a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-09-08T16:36:01.949 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127425", "LimitNPROCSoft": "127425", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127425", "LimitSIGPENDINGSoft": "127425", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2500", "MemoryAccounting": "yes", "MemoryCurrent": "1171456", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-09-08T16:36:01.961 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2020-09-08 16:31:35 UTC", "StateChangeTimestampMonotonic": "16071774", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2020-09-08 16:31:35 UTC", "WatchdogTimestampMonotonic": "16071773", "WatchdogUSec": "0"}} 2020-09-08T16:36:02.000 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:01 +0000 (0:00:00.918) 0:00:28.418 ***** 2020-09-08T16:36:02.213 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-09-08T16:36:02.215 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001713", "end": "2020-09-08 16:36:02.190747", "rc": 0, "start": "2020-09-08 16:36:02.189034", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-09-08T16:36:02.256 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:02 +0000 (0:00:00.255) 0:00:28.674 ***** 2020-09-08T16:36:03.247 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-09-08T16:36:03.303 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:03 +0000 (0:00:01.047) 0:00:29.721 ***** 2020-09-08T16:36:04.273 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-09-08T16:36:04.275 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-09-08T16:36:04.310 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:04 +0000 (0:00:01.007) 0:00:30.728 ***** 2020-09-08T16:36:05.886 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-09-08T16:36:05.925 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:05 +0000 (0:00:01.611) 0:00:32.340 ***** 2020-09-08T16:36:22.368 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-09-08T16:36:22.370 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.237174", "end": "2020-09-08 16:36:22.347749", "failed_when_result": false, "rc": 0, "start": "2020-09-08 16:36:06.110575", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-09-08T16:36:22.408 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:22 +0000 (0:00:16.486) 0:00:48.826 ***** 2020-09-08T16:36:22.988 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-09-08T16:36:22.990 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-09-08T16:36:23.027 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:23 +0000 (0:00:00.618) 0:00:49.445 ***** 2020-09-08T16:36:23.270 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-09-08T16:36:23.272 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015463", "end": "2020-09-08 16:36:23.255479", "rc": 0, "start": "2020-09-08 16:36:23.240016", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:36:23.326 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:23 +0000 (0:00:00.298) 0:00:49.744 ***** 2020-09-08T16:36:23.545 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-09-08T16:36:23.547 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.009788", "end": "2020-09-08 16:36:23.532336", "rc": 0, "start": "2020-09-08 16:36:23.522548", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:36:23.584 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:23 +0000 (0:00:00.258) 0:00:50.002 ***** 2020-09-08T16:36:39.969 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-09-08T16:36:39.972 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.163504", "end": "2020-09-08 16:36:39.946192", "rc": 0, "start": "2020-09-08 16:36:23.782688", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:36:40.007 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:40 +0000 (0:00:16.422) 0:01:06.425 ***** 2020-09-08T16:36:40.235 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-09-08T16:36:40.237 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-09-08T16:36:40.275 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:40 +0000 (0:00:00.267) 0:01:06.693 ***** 2020-09-08T16:36:40.316 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-09-08T16:36:40.318 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-09-08T16:36:40.354 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:40 +0000 (0:00:00.075) 0:01:06.769 ***** 2020-09-08T16:36:40.392 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-09-08T16:36:40.394 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-09-08T16:36:40.428 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:40 +0000 (0:00:00.076) 0:01:06.846 ***** 2020-09-08T16:36:40.485 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-09-08T16:36:40.487 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-09-08T16:36:40.522 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:40 +0000 (0:00:00.094) 0:01:06.940 ***** 2020-09-08T16:36:40.575 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi120.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-09-08T16:36:40.611 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:40 +0000 (0:00:00.088) 0:01:07.029 ***** 2020-09-08T16:36:40.809 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-09-08T16:36:40.811 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-09-08T16:36:40.863 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:40 +0000 (0:00:00.240) 0:01:07.270 ***** 2020-09-08T16:36:41.069 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1229, "name": "ubuntu", "state": "present", "system": false} 2020-09-08T16:36:41.107 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:41 +0000 (0:00:00.255) 0:01:07.525 ***** 2020-09-08T16:36:41.139 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-09-08T16:36:41.141 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:41.178 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:41 +0000 (0:00:00.071) 0:01:07.596 ***** 2020-09-08T16:36:41.210 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-09-08T16:36:41.212 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:41.245 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:41 +0000 (0:00:00.066) 0:01:07.663 ***** 2020-09-08T16:36:41.475 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi120.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-09-08T16:36:41.517 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:41 +0000 (0:00:00.266) 0:01:07.929 ***** 2020-09-08T16:36:42.072 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-09-08T16:36:42.074 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-09-08T16:36:42.112 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:42 +0000 (0:00:00.596) 0:01:08.526 ***** 2020-09-08T16:36:42.148 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-09-08T16:36:42.150 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi120"}, "changed": false} 2020-09-08T16:36:42.183 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:42 +0000 (0:00:00.075) 0:01:08.601 ***** 2020-09-08T16:36:43.322 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-09-08T16:36:43.324 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi120", "ansible_hostname": "smithi120", "ansible_nodename": "smithi120"}, "changed": false, "name": "smithi120"} 2020-09-08T16:36:43.358 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:43 +0000 (0:00:01.174) 0:01:09.776 ***** 2020-09-08T16:36:43.852 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-09-08T16:36:43.854 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-09-08T16:36:43.906 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:43 +0000 (0:00:00.547) 0:01:10.323 ***** 2020-09-08T16:36:44.438 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-09-08T16:36:44.440 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-09-08T16:36:44.474 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:44 +0000 (0:00:00.568) 0:01:10.892 ***** 2020-09-08T16:36:45.344 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-09-08T16:36:45.346 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-09-08T16:36:45.378 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:45 +0000 (0:00:00.903) 0:01:11.796 ***** 2020-09-08T16:36:45.892 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-09-08T16:36:45.895 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-09-08T16:36:45.927 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:45 +0000 (0:00:00.549) 0:01:12.345 ***** 2020-09-08T16:36:46.149 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-09-08T16:36:46.151 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-09-08T16:36:46.190 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:46 +0000 (0:00:00.262) 0:01:12.607 ***** 2020-09-08T16:36:46.409 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-09-08T16:36:46.442 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:46 +0000 (0:00:00.252) 0:01:12.860 ***** 2020-09-08T16:36:46.674 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi120.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1229, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-09-08T16:36:46.708 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:46 +0000 (0:00:00.265) 0:01:13.126 ***** 2020-09-08T16:36:47.223 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-09-08T16:36:47.265 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:47 +0000 (0:00:00.557) 0:01:13.683 ***** 2020-09-08T16:36:47.776 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-09-08T16:36:47.778 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-09-08T16:36:47.811 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:47 +0000 (0:00:00.546) 0:01:14.229 ***** 2020-09-08T16:36:49.779 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-09-08T16:36:49.782 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.746611", "end": "2020-09-08 16:36:49.757624", "rc": 0, "start": "2020-09-08 16:36:48.011013", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:36:49.817 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:49 +0000 (0:00:02.005) 0:01:16.235 ***** 2020-09-08T16:36:50.353 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** ok: [smithi120.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2020-09-08T16:36:50.842 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2020-09-08T16:36:51.329 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2020-09-08T16:36:51.370 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:51 +0000 (0:00:01.552) 0:01:17.788 ***** 2020-09-08T16:36:51.889 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-09-08T16:36:51.891 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-09-08T16:36:51.934 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:51 +0000 (0:00:00.563) 0:01:18.352 ***** 2020-09-08T16:36:52.463 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-09-08T16:36:52.465 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-09-08T16:36:52.944 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-09-08T16:36:53.434 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-09-08T16:36:53.472 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:53 +0000 (0:00:01.538) 0:01:19.890 ***** 2020-09-08T16:36:53.503 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-09-08T16:36:53.538 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:53 +0000 (0:00:00.065) 0:01:19.956 ***** 2020-09-08T16:36:53.579 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-09-08T16:36:53.580 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:53.615 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:53 +0000 (0:00:00.076) 0:01:20.033 ***** 2020-09-08T16:36:53.710 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi120.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi120.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-09-08T16:36:53.717 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:53 +0000 (0:00:00.101) 0:01:20.135 ***** 2020-09-08T16:36:53.758 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-09-08T16:36:53.760 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:53.807 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:53 +0000 (0:00:00.090) 0:01:20.225 ***** 2020-09-08T16:36:53.851 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-09-08T16:36:53.853 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-09-08T16:36:53.896 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:53 +0000 (0:00:00.089) 0:01:20.314 ***** 2020-09-08T16:36:53.940 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-09-08T16:36:53.942 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:54.008 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:54 +0000 (0:00:00.111) 0:01:20.426 ***** 2020-09-08T16:36:54.062 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-09-08T16:36:54.065 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:54.096 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:54 +0000 (0:00:00.088) 0:01:20.514 ***** 2020-09-08T16:36:54.340 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-09-08T16:36:54.342 INFO:teuthology.task.ansible.out:fatal: [smithi120.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006898", "end": "2020-09-08 16:36:54.319651", "msg": "non-zero return code", "rc": 1, "start": "2020-09-08 16:36:54.312753", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-09-08T16:36:54.384 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:54 +0000 (0:00:00.287) 0:01:20.802 ***** 2020-09-08T16:36:54.426 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-09-08T16:36:54.428 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:36:54.461 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:54 +0000 (0:00:00.077) 0:01:20.879 ***** 2020-09-08T16:36:56.838 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-09-08T16:36:56.840 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-09-08T16:36:56.877 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:56 +0000 (0:00:02.415) 0:01:23.295 ***** 2020-09-08T16:36:58.889 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-09-08T16:36:58.891 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2020-09-08T16:36:58.923 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:36:58 +0000 (0:00:02.046) 0:01:25.341 ***** 2020-09-08T16:37:01.967 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-09-08T16:37:01.969 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64"]} 2020-09-08T16:37:02.004 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:02 +0000 (0:00:03.080) 0:01:28.422 ***** 2020-09-08T16:37:02.044 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-09-08T16:37:02.046 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:02.082 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:02 +0000 (0:00:00.077) 0:01:28.500 ***** 2020-09-08T16:37:03.026 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-09-08T16:37:03.028 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-09-08T16:37:03.063 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:03 +0000 (0:00:00.981) 0:01:29.481 ***** 2020-09-08T16:37:03.106 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-09-08T16:37:03.108 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:03.141 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:03 +0000 (0:00:00.077) 0:01:29.559 ***** 2020-09-08T16:37:03.486 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-09-08T16:37:03.487 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service sysinit.target basic.target polkit.service system.slice dbus.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service ip6tables.service iptables.service shutdown.target ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-09-08T16:37:03.489 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127425", "LimitNPROCSoft": "127425", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127425", 2020-09-08T16:37:03.491 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127425", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-09-08T16:37:03.492 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-09-08T16:37:03.523 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:03 +0000 (0:00:00.381) 0:01:29.941 ***** 2020-09-08T16:37:03.827 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-09-08T16:37:03.829 INFO:teuthology.task.ansible.out:fatal: [smithi120.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-09-08T16:37:03.863 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:03 +0000 (0:00:00.340) 0:01:30.281 ***** 2020-09-08T16:37:04.693 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-09-08T16:37:04.695 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-09-08T16:37:04.728 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:04 +0000 (0:00:00.865) 0:01:31.146 ***** 2020-09-08T16:37:04.762 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:04.795 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:04 +0000 (0:00:00.066) 0:01:31.213 ***** 2020-09-08T16:37:04.826 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-09-08T16:37:04.828 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:04.861 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:04 +0000 (0:00:00.065) 0:01:31.279 ***** 2020-09-08T16:37:04.894 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-09-08T16:37:04.895 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:04.930 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:04 +0000 (0:00:00.068) 0:01:31.348 ***** 2020-09-08T16:37:04.960 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-09-08T16:37:04.962 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:04.997 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:04 +0000 (0:00:00.067) 0:01:31.415 ***** 2020-09-08T16:37:05.027 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-09-08T16:37:05.029 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.062 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.065) 0:01:31.480 ***** 2020-09-08T16:37:05.093 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-09-08T16:37:05.095 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.129 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:31.547 ***** 2020-09-08T16:37:05.159 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-09-08T16:37:05.194 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.065) 0:01:31.612 ***** 2020-09-08T16:37:05.224 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-09-08T16:37:05.226 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.260 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:31.678 ***** 2020-09-08T16:37:05.293 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-09-08T16:37:05.295 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.329 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.068) 0:01:31.747 ***** 2020-09-08T16:37:05.370 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-09-08T16:37:05.372 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.381 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.417 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.088) 0:01:31.835 ***** 2020-09-08T16:37:05.448 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-09-08T16:37:05.450 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.487 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.069) 0:01:31.905 ***** 2020-09-08T16:37:05.518 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-09-08T16:37:05.553 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:31.971 ***** 2020-09-08T16:37:05.584 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-09-08T16:37:05.585 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.619 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.065) 0:01:32.037 ***** 2020-09-08T16:37:05.650 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-09-08T16:37:05.651 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.685 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:32.103 ***** 2020-09-08T16:37:05.716 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-09-08T16:37:05.718 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.752 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:32.170 ***** 2020-09-08T16:37:05.782 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-09-08T16:37:05.784 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.818 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:32.236 ***** 2020-09-08T16:37:05.850 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-09-08T16:37:05.851 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.885 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:32.303 ***** 2020-09-08T16:37:05.918 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-09-08T16:37:05.920 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:05.952 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:05 +0000 (0:00:00.066) 0:01:32.370 ***** 2020-09-08T16:37:05.983 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-09-08T16:37:05.985 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.018 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.065) 0:01:32.436 ***** 2020-09-08T16:37:06.059 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-09-08T16:37:06.060 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.093 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.074) 0:01:32.511 ***** 2020-09-08T16:37:06.123 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-09-08T16:37:06.125 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.157 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.064) 0:01:32.575 ***** 2020-09-08T16:37:06.197 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-09-08T16:37:06.198 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.232 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.074) 0:01:32.650 ***** 2020-09-08T16:37:06.261 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-09-08T16:37:06.262 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.295 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.063) 0:01:32.713 ***** 2020-09-08T16:37:06.336 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-09-08T16:37:06.338 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.369 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.074) 0:01:32.787 ***** 2020-09-08T16:37:06.398 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-09-08T16:37:06.400 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.433 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.063) 0:01:32.851 ***** 2020-09-08T16:37:06.473 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-09-08T16:37:06.475 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.510 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.076) 0:01:32.928 ***** 2020-09-08T16:37:06.539 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-09-08T16:37:06.540 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.573 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.062) 0:01:32.991 ***** 2020-09-08T16:37:06.622 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-09-08T16:37:06.624 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.658 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.083) 0:01:33.075 ***** 2020-09-08T16:37:06.687 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-09-08T16:37:06.688 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.721 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.064) 0:01:33.139 ***** 2020-09-08T16:37:06.763 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-09-08T16:37:06.764 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.796 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.074) 0:01:33.214 ***** 2020-09-08T16:37:06.826 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-09-08T16:37:06.828 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.861 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.065) 0:01:33.279 ***** 2020-09-08T16:37:06.891 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-09-08T16:37:06.893 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.926 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.065) 0:01:33.344 ***** 2020-09-08T16:37:06.955 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-09-08T16:37:06.957 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:06.989 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:06 +0000 (0:00:00.063) 0:01:33.407 ***** 2020-09-08T16:37:07.030 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-09-08T16:37:07.032 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:07.065 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.075) 0:01:33.483 ***** 2020-09-08T16:37:07.095 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-09-08T16:37:07.096 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:07.130 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.065) 0:01:33.548 ***** 2020-09-08T16:37:07.171 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-09-08T16:37:07.173 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:07.204 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.074) 0:01:33.622 ***** 2020-09-08T16:37:07.234 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-09-08T16:37:07.236 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:07.270 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.066) 0:01:33.688 ***** 2020-09-08T16:37:07.301 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-09-08T16:37:07.303 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:07.337 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.066) 0:01:33.755 ***** 2020-09-08T16:37:07.368 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-09-08T16:37:07.369 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:07.403 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.065) 0:01:33.821 ***** 2020-09-08T16:37:07.642 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-09-08T16:37:07.644 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-09-08T16:37:07.676 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.273) 0:01:34.094 ***** 2020-09-08T16:37:07.913 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-09-08T16:37:07.915 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-09-08T16:37:07.949 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:07 +0000 (0:00:00.272) 0:01:34.367 ***** 2020-09-08T16:37:08.164 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-09-08T16:37:08.166 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-09-08T16:37:08.200 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.251) 0:01:34.618 ***** 2020-09-08T16:37:08.229 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-09-08T16:37:08.231 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.264 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.063) 0:01:34.682 ***** 2020-09-08T16:37:08.294 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-09-08T16:37:08.296 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.329 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.065) 0:01:34.747 ***** 2020-09-08T16:37:08.358 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-09-08T16:37:08.359 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.392 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.063) 0:01:34.810 ***** 2020-09-08T16:37:08.423 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-09-08T16:37:08.425 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.457 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.065) 0:01:34.875 ***** 2020-09-08T16:37:08.486 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-09-08T16:37:08.488 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.520 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.063) 0:01:34.938 ***** 2020-09-08T16:37:08.550 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-09-08T16:37:08.551 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.584 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.063) 0:01:35.002 ***** 2020-09-08T16:37:08.612 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-09-08T16:37:08.614 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.648 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.064) 0:01:35.066 ***** 2020-09-08T16:37:08.689 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-09-08T16:37:08.690 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.723 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.075) 0:01:35.141 ***** 2020-09-08T16:37:08.753 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-09-08T16:37:08.754 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.787 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.063) 0:01:35.205 ***** 2020-09-08T16:37:08.818 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-09-08T16:37:08.820 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.851 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.064) 0:01:35.269 ***** 2020-09-08T16:37:08.890 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-09-08T16:37:08.891 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.927 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.075) 0:01:35.345 ***** 2020-09-08T16:37:08.956 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-09-08T16:37:08.958 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:08.991 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:08 +0000 (0:00:00.064) 0:01:35.409 ***** 2020-09-08T16:37:09.020 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-09-08T16:37:09.022 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.055 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.063) 0:01:35.473 ***** 2020-09-08T16:37:09.095 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-09-08T16:37:09.097 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.132 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.076) 0:01:35.550 ***** 2020-09-08T16:37:09.163 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-09-08T16:37:09.165 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.201 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.068) 0:01:35.619 ***** 2020-09-08T16:37:09.250 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.296 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.094) 0:01:35.714 ***** 2020-09-08T16:37:09.338 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-09-08T16:37:09.340 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.379 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.083) 0:01:35.797 ***** 2020-09-08T16:37:09.410 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-09-08T16:37:09.411 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.450 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.071) 0:01:35.868 ***** 2020-09-08T16:37:09.482 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-09-08T16:37:09.484 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.522 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.071) 0:01:35.940 ***** 2020-09-08T16:37:09.554 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-09-08T16:37:09.556 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.589 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.067) 0:01:36.007 ***** 2020-09-08T16:37:09.620 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-09-08T16:37:09.622 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.654 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.065) 0:01:36.072 ***** 2020-09-08T16:37:09.686 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-09-08T16:37:09.688 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.720 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.065) 0:01:36.138 ***** 2020-09-08T16:37:09.761 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-09-08T16:37:09.763 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-09-08T16:37:09.795 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.075) 0:01:36.213 ***** 2020-09-08T16:37:09.846 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-09-08T16:37:09.848 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.858 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-09-08T16:37:09.868 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.870 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.879 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.881 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.890 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:09.934 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:09 +0000 (0:00:00.138) 0:01:36.352 ***** 2020-09-08T16:37:10.161 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-09-08T16:37:10.163 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004624", "end": "2020-09-08 16:37:10.132838", "rc": 0, "start": "2020-09-08 16:37:10.128214", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-09-08T16:37:10.200 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:10 +0000 (0:00:00.266) 0:01:36.618 ***** 2020-09-08T16:37:10.240 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-09-08T16:37:10.242 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:10.280 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:10 +0000 (0:00:00.079) 0:01:36.698 ***** 2020-09-08T16:37:10.322 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-09-08T16:37:10.324 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:10.358 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:10 +0000 (0:00:00.077) 0:01:36.776 ***** 2020-09-08T16:37:10.394 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-09-08T16:37:10.396 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:10.432 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:10 +0000 (0:00:00.072) 0:01:36.849 ***** 2020-09-08T16:37:10.460 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-09-08T16:37:10.462 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:10.497 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:10 +0000 (0:00:00.066) 0:01:36.915 ***** 2020-09-08T16:37:11.444 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-09-08T16:37:11.445 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-09-08T16:37:11.499 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:11 +0000 (0:00:01.001) 0:01:37.917 ***** 2020-09-08T16:37:11.553 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-09-08T16:37:11.555 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930530324480, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227180255, 'block_used': 12914853, 'inode_total': 61054976, 'inode_available': 60990174, 'inode_used': 64802, 'uuid': '7e53389d-f8ef-4bef-b3f0-509d591fc1e4'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227180255, "block_size": 4096, "block_total": 240095108, "block_used": 12914853, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990174, "inode_total": 61054976, "inode_used": 64802, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930530324480, "size_total": 983429562368, "uuid": "7e53389d-f8ef-4bef-b3f0-509d591fc1e4"}} 2020-09-08T16:37:11.609 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:11 +0000 (0:00:00.109) 0:01:38.027 ***** 2020-09-08T16:37:11.828 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-09-08T16:37:11.830 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.004968", "end": "2020-09-08 16:37:11.807799", "rc": 0, "start": "2020-09-08 16:37:11.802831", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:11.875 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:11 +0000 (0:00:00.266) 0:01:38.293 ***** 2020-09-08T16:37:11.928 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-09-08T16:37:11.930 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930530324480, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227180255, 'block_used': 12914853, 'inode_total': 61054976, 'inode_available': 60990174, 'inode_used': 64802, 'uuid': '7e53389d-f8ef-4bef-b3f0-509d591fc1e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227180255, "block_size": 4096, "block_total": 240095108, "block_used": 12914853, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990174, "inode_total": 61054976, "inode_used": 64802, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930530324480, "size_total": 983429562368, "uuid": "7e53389d-f8ef-4bef-b3f0-509d591fc1e4"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:11.979 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:11 +0000 (0:00:00.104) 0:01:38.397 ***** 2020-09-08T16:37:12.010 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-09-08T16:37:12.046 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:12 +0000 (0:00:00.066) 0:01:38.464 ***** 2020-09-08T16:37:12.047 INFO:teuthology.task.ansible.out: 2020-09-08T16:37:13.631 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-09-08T16:37:13.633 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyhpZItg2ooRQxIAdhi5MUcvPnvwGQMIDk'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.341833", "end": "2020-09-08 16:37:13.600369", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyhpZItg2ooRQxIAdhi5MUcvPnvwGQMIDk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:12.258536", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-09-08T16:37:14.955 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-7MhrcJ-4GU9-VPpv-JmVa-qpc2-eEQs-cGdESg', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN', 'nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.102333", "end": "2020-09-08 16:37:14.925142", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7MhrcJ-4GU9-VPpv-JmVa-qpc2-eEQs-cGdESg", "nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN", "nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:13.822809", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-09-08T16:37:14.957 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-09-08T16:37:16.265 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyGARPbfwADbUDLQoUXvzAB8IySw7rPaCb'], 'uuids': ['3041ed33-1b71-4177-bd18-23f03c8d675c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.100499", "end": "2020-09-08 16:37:16.237402", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyGARPbfwADbUDLQoUXvzAB8IySw7rPaCb"], "labels": [], "masters": [], "uuids": ["3041ed33-1b71-4177-bd18-23f03c8d675c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:15.136903", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-09-08T16:37:17.558 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyKrYGD2xcYQcsleGQDwSaDz4IPAeogaRv'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.081801", "end": "2020-09-08 16:37:17.531489", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyKrYGD2xcYQcsleGQDwSaDz4IPAeogaRv"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:16.449688", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-09-08T16:37:18.819 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvynCKsp04ME2ubOmCdLNTeYJKIb5TxlTo0'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.060911", "end": "2020-09-08 16:37:18.791708", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvynCKsp04ME2ubOmCdLNTeYJKIb5TxlTo0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:17.730797", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-09-08T16:37:18.821 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0', 'wwn-0x5000c50091e1af53'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QHP0', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e1af53', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1', 'wwn-0x5000c50091e1af53-part1'], 'uuids': ['7e53389d-f8ef-4bef-b3f0-509d591fc1e4'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '7e53389d-f8ef-4bef-b3f0-509d591fc1e4', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0", "wwn-0x5000c50091e1af53"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1", "wwn-0x5000c50091e1af53-part1"], "labels": [], "masters": [], "uuids": ["7e53389d-f8ef-4bef-b3f0-509d591fc1e4"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "7e53389d-f8ef-4bef-b3f0-509d591fc1e4"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHP0", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1af53"}}, 2020-09-08T16:37:18.823 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-09-08T16:37:20.100 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyP1wKitL0QDKNG7atK06QxEOqxwVCGZwe'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.070458", "end": "2020-09-08 16:37:20.072420", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyP1wKitL0QDKNG7atK06QxEOqxwVCGZwe"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:19.001962", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-09-08T16:37:20.140 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:20 +0000 (0:00:08.094) 0:01:46.558 ***** 2020-09-08T16:37:20.532 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-09-08T16:37:20.534 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyhpZItg2ooRQxIAdhi5MUcvPnvwGQMIDk'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139054", "end": "2020-09-08 16:37:20.505031", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyhpZItg2ooRQxIAdhi5MUcvPnvwGQMIDk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:20.365977", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136456 s, 845 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136456 s, 845 MB/s"], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:20.758 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-7MhrcJ-4GU9-VPpv-JmVa-qpc2-eEQs-cGdESg', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN', 'nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036997", "end": "2020-09-08 16:37:20.732373", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7MhrcJ-4GU9-VPpv-JmVa-qpc2-eEQs-cGdESg", "nvme-INTEL_SSDPEDMD400G4_PHFT620400XV400BGN", "nvme-nvme.8086-50484654363230343030585634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:20.695376", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.034413 s, 3.4 GB/s", "stde 2020-09-08T16:37:20.760 INFO:teuthology.task.ansible.out:rr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.034413 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:21.097 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyGARPbfwADbUDLQoUXvzAB8IySw7rPaCb'], 'uuids': ['3041ed33-1b71-4177-bd18-23f03c8d675c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148302", "end": "2020-09-08 16:37:21.079301", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyGARPbfwADbUDLQoUXvzAB8IySw7rPaCb"], "labels": [], "masters": [], "uuids": ["3041ed33-1b71-4177-bd18-23f03c8d675c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:20.930999", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145772 s, 791 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145772 s, 791 MB/s"], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:21.429 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyKrYGD2xcYQcsleGQDwSaDz4IPAeogaRv'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.140373", "end": "2020-09-08 16:37:21.410421", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyKrYGD2xcYQcsleGQDwSaDz4IPAeogaRv"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:21.270048", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137818 s, 837 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137818 s, 837 MB/s"], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:21.761 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvynCKsp04ME2ubOmCdLNTeYJKIb5TxlTo0'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.139078", "end": "2020-09-08 16:37:21.737095", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvynCKsp04ME2ubOmCdLNTeYJKIb5TxlTo0"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:21.598017", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136607 s, 844 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136607 s, 844 MB/s"], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:21.762 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0', 'wwn-0x5000c50091e1af53'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QHP0', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e1af53', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1', 'wwn-0x5000c50091e1af53-part1'], 'uuids': ['7e53389d-f8ef-4bef-b3f0-509d591fc1e4'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '7e53389d-f8ef-4bef-b3f0-509d591fc1e4', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0", "wwn-0x5000c50091e1af53"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHP0-part1", "wwn-0x5000c50091e1af53-part1"], "labels": [], "masters": [], "uuids": ["7e53389d-f8ef-4bef-b3f0-509d591fc1e4"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "7e53389d-f8ef-4bef-b3f0-509d591fc1e4"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHP0", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1af53"}}, 2020-09-08T16:37:21.764 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2020-09-08T16:37:22.091 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyP1wKitL0QDKNG7atK06QxEOqxwVCGZwe'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.138676", "end": "2020-09-08 16:37:22.072002", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-fjl5r8ilfZC8WlC9z56i0ZZo1RDKwmvyP1wKitL0QDKNG7atK06QxEOqxwVCGZwe"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-09-08 16:37:21.933326", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136066 s, 848 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136066 s, 848 MB/s"], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:22.133 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:22 +0000 (0:00:01.992) 0:01:48.551 ***** 2020-09-08T16:37:22.420 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-09-08T16:37:22.422 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.070929", "end": "2020-09-08 16:37:22.395994", "rc": 0, "start": "2020-09-08 16:37:22.325065", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:22.456 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:22 +0000 (0:00:00.323) 0:01:48.874 ***** 2020-09-08T16:37:22.721 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-09-08T16:37:22.723 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049816", "end": "2020-09-08 16:37:22.704953", "rc": 0, "start": "2020-09-08 16:37:22.655137", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:22.759 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:22 +0000 (0:00:00.302) 0:01:49.177 ***** 2020-09-08T16:37:22.790 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-09-08T16:37:22.829 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:22 +0000 (0:00:00.070) 0:01:49.247 ***** 2020-09-08T16:37:22.860 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-09-08T16:37:22.862 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:22.899 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:22 +0000 (0:00:00.069) 0:01:49.317 ***** 2020-09-08T16:37:22.929 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-09-08T16:37:22.931 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:22.966 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:22 +0000 (0:00:00.067) 0:01:49.384 ***** 2020-09-08T16:37:22.999 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-09-08T16:37:23.000 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.045 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.078) 0:01:49.463 ***** 2020-09-08T16:37:23.081 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-09-08T16:37:23.083 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.117 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.072) 0:01:49.535 ***** 2020-09-08T16:37:23.337 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-09-08T16:37:23.338 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-09-08T16:37:23.376 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.259) 0:01:49.794 ***** 2020-09-08T16:37:23.429 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-09-08T16:37:23.431 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930530324480, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227180255, 'block_used': 12914853, 'inode_total': 61054976, 'inode_available': 60990174, 'inode_used': 64802, 'uuid': '7e53389d-f8ef-4bef-b3f0-509d591fc1e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227180255, "block_size": 4096, "block_total": 240095108, "block_used": 12914853, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60990174, "inode_total": 61054976, "inode_used": 64802, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930530324480, "size_total": 983429562368, "uuid": "7e53389d-f8ef-4bef-b3f0-509d591fc1e4"}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.466 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.089) 0:01:49.884 ***** 2020-09-08T16:37:23.499 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-09-08T16:37:23.501 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.541 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.075) 0:01:49.959 ***** 2020-09-08T16:37:23.585 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-09-08T16:37:23.587 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.628 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.086) 0:01:50.046 ***** 2020-09-08T16:37:23.674 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-09-08T16:37:23.676 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.710 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.081) 0:01:50.128 ***** 2020-09-08T16:37:23.751 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-09-08T16:37:23.753 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.785 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.075) 0:01:50.203 ***** 2020-09-08T16:37:23.837 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-09-08T16:37:23.839 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.872 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.086) 0:01:50.290 ***** 2020-09-08T16:37:23.914 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-09-08T16:37:23.915 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:23.949 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:23 +0000 (0:00:00.077) 0:01:50.367 ***** 2020-09-08T16:37:24.731 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-09-08T16:37:24.733 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-09-08T16:37:24.766 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:24 +0000 (0:00:00.816) 0:01:51.183 ***** 2020-09-08T16:37:25.524 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-09-08T16:37:25.526 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-09-08T16:37:25.877 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-09-08T16:37:26.222 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-09-08T16:37:26.564 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-09-08T16:37:26.917 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-09-08T16:37:26.954 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:26 +0000 (0:00:02.188) 0:01:53.372 ***** 2020-09-08T16:37:27.162 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-09-08T16:37:27.164 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-09-08T16:37:27.202 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:27 +0000 (0:00:00.248) 0:01:53.620 ***** 2020-09-08T16:37:27.450 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-09-08T16:37:27.452 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-09-08T16:37:27.634 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-09-08T16:37:27.835 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-09-08T16:37:28.038 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-09-08T16:37:28.055 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:28.093 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:28 +0000 (0:00:00.890) 0:01:54.511 ***** 2020-09-08T16:37:28.311 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-09-08T16:37:28.313 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-09-08T16:37:28.350 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:28 +0000 (0:00:00.257) 0:01:54.768 ***** 2020-09-08T16:37:28.395 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-09-08T16:37:28.397 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-09-08T16:37:28.434 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:28 +0000 (0:00:00.083) 0:01:54.851 ***** 2020-09-08T16:37:29.043 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-09-08T16:37:29.045 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true} 2020-09-08T16:37:29.085 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:29 +0000 (0:00:00.650) 0:01:55.502 ***** 2020-09-08T16:37:29.767 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-09-08T16:37:29.769 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-09-08T16:37:29.802 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:29 +0000 (0:00:00.717) 0:01:56.220 ***** 2020-09-08T16:37:29.831 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-09-08T16:37:29.833 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:29.865 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:29 +0000 (0:00:00.063) 0:01:56.283 ***** 2020-09-08T16:37:29.895 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-09-08T16:37:29.897 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:29.931 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:29 +0000 (0:00:00.065) 0:01:56.349 ***** 2020-09-08T16:37:29.960 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-09-08T16:37:29.962 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:29.997 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:29 +0000 (0:00:00.065) 0:01:56.415 ***** 2020-09-08T16:37:30.025 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-09-08T16:37:30.027 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.059 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.062) 0:01:56.477 ***** 2020-09-08T16:37:30.089 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-09-08T16:37:30.091 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.122 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.063) 0:01:56.540 ***** 2020-09-08T16:37:30.150 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-09-08T16:37:30.153 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.185 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.062) 0:01:56.603 ***** 2020-09-08T16:37:30.212 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-09-08T16:37:30.214 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.246 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.061) 0:01:56.664 ***** 2020-09-08T16:37:30.273 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-09-08T16:37:30.275 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.309 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.061) 0:01:56.725 ***** 2020-09-08T16:37:30.336 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.370 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.062) 0:01:56.788 ***** 2020-09-08T16:37:30.398 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-09-08T16:37:30.400 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.432 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.062) 0:01:56.850 ***** 2020-09-08T16:37:30.460 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-09-08T16:37:30.462 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:30.493 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:30 +0000 (0:00:00.061) 0:01:56.911 ***** 2020-09-08T16:37:30.988 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-09-08T16:37:30.990 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-09-08T16:37:31.022 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:31 +0000 (0:00:00.529) 0:01:57.440 ***** 2020-09-08T16:37:31.346 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-09-08T16:37:31.348 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2020-09-08 16:31:35 UTC", "ActiveEnterTimestampMonotonic": "16118219", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpd.service basic.target sysinit.target sntp.service -.mount systemd-tmpfiles-setup.service tmp.mount system.slice ntpdate.service systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2020-09-08 16:31:34 UTC", "AssertTimestampMonotonic": "15082701", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2020-09-08 16:31:34 UTC", "ConditionTimestampMonotonic": "15082674", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-09-08T16:37:31.349 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2461", "ExecMainStartTimestamp": "Tue 2020-09-08 16:31:35 UTC", "ExecMainStartTimestampMonotonic": "16041092", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[Tue 2020-09-08 16:31:34 UTC] ; stop_time=[Tue 2020-09-08 16:31:35 UTC] ; pid=2453 ; code=exited ; status=0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[Tue 2020-09-08 16:31:35 UTC] ; stop_time=[Tue 2020-09-08 16:31:35 UTC] ; pid=2496 ; code=exited ; status=0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2020-09-08 16:31:34 UTC", "InactiveExitTimestampMonotonic": "15132436", "InvocationID": "8f837e6e9e5049b8a9051e4a7ba87088", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZE 2020-09-08T16:37:31.351 INFO:teuthology.task.ansible.out:Soft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127425", "LimitNPROCSoft": "127425", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127425", "LimitSIGPENDINGSoft": "127425", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2461", "MemoryAccounting": "yes", "MemoryCurrent": "2445312", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no" 2020-09-08T16:37:31.353 INFO:teuthology.task.ansible.out:, "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2020-09-08 16:31:35 UTC", "StateChangeTimestampMonotonic": "16118219", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2020-09-08 16:31:35 UTC", "WatchdogTimestampMonotonic": "16041104", "WatchdogUSec": "0"}} 2020-09-08T16:37:31.382 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:31 +0000 (0:00:00.360) 0:01:57.800 ***** 2020-09-08T16:37:31.413 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-09-08T16:37:31.415 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-09-08T16:37:31.448 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:31 +0000 (0:00:00.065) 0:01:57.866 ***** 2020-09-08T16:37:31.489 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-09-08T16:37:31.491 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-09-08T16:37:31.530 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:31 +0000 (0:00:00.082) 0:01:57.948 ***** 2020-09-08T16:37:31.561 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-09-08T16:37:31.563 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:31.596 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:31 +0000 (0:00:00.066) 0:01:58.014 ***** 2020-09-08T16:37:31.640 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-09-08T16:37:31.642 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:31.674 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:31 +0000 (0:00:00.078) 0:01:58.092 ***** 2020-09-08T16:37:31.717 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-09-08T16:37:31.719 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:31.753 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:31 +0000 (0:00:00.078) 0:01:58.171 ***** 2020-09-08T16:37:31.970 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-09-08T16:37:31.972 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "gid": 1229, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-09-08T16:37:32.005 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:32 +0000 (0:00:00.252) 0:01:58.423 ***** 2020-09-08T16:37:32.525 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-09-08T16:37:32.527 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1229, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-09-08T16:37:32.560 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:32 +0000 (0:00:00.555) 0:01:58.978 ***** 2020-09-08T16:37:33.964 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-09-08T16:37:33.966 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-09-08T16:37:34.005 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:01.444) 0:02:00.423 ***** 2020-09-08T16:37:34.036 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-09-08T16:37:34.038 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:34.070 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:00.065) 0:02:00.488 ***** 2020-09-08T16:37:34.111 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-09-08T16:37:34.113 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-09-08T16:37:34.147 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:00.076) 0:02:00.565 ***** 2020-09-08T16:37:34.342 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-09-08T16:37:34.344 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-09-08T16:37:34.377 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:00.229) 0:02:00.795 ***** 2020-09-08T16:37:34.413 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-09-08T16:37:34.414 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:34.449 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:00.072) 0:02:00.867 ***** 2020-09-08T16:37:34.490 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-09-08T16:37:34.492 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-09-08T16:37:34.526 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:00.076) 0:02:00.944 ***** 2020-09-08T16:37:34.557 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:34.595 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:00.068) 0:02:01.013 ***** 2020-09-08T16:37:34.811 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-09-08T16:37:34.812 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-09-08T16:37:34.847 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:34 +0000 (0:00:00.252) 0:02:01.265 ***** 2020-09-08T16:37:35.053 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-09-08T16:37:35.055 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-09-08T16:37:35.094 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.246) 0:02:01.512 ***** 2020-09-08T16:37:35.126 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-09-08T16:37:35.127 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.164 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.070) 0:02:01.582 ***** 2020-09-08T16:37:35.207 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-09-08T16:37:35.208 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.210 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.221 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.222 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.224 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.259 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.094) 0:02:01.677 ***** 2020-09-08T16:37:35.303 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-09-08T16:37:35.304 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.306 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.316 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.318 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.329 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.365 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.106) 0:02:01.783 ***** 2020-09-08T16:37:35.398 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.439 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.074) 0:02:01.857 ***** 2020-09-08T16:37:35.472 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-09-08T16:37:35.474 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.509 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.069) 0:02:01.927 ***** 2020-09-08T16:37:35.540 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-09-08T16:37:35.541 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.581 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.072) 0:02:01.999 ***** 2020-09-08T16:37:35.617 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-09-08T16:37:35.618 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.654 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.072) 0:02:02.072 ***** 2020-09-08T16:37:35.685 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-09-08T16:37:35.686 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.724 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.069) 0:02:02.141 ***** 2020-09-08T16:37:35.754 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-09-08T16:37:35.756 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.790 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.066) 0:02:02.208 ***** 2020-09-08T16:37:35.832 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-09-08T16:37:35.834 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.870 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.080) 0:02:02.288 ***** 2020-09-08T16:37:35.901 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-09-08T16:37:35.903 INFO:teuthology.task.ansible.out:skipping: [smithi120.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-09-08T16:37:35.936 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:35 +0000 (0:00:00.065) 0:02:02.354 ***** 2020-09-08T16:37:36.175 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-09-08T16:37:36.177 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004838", "end": "2020-09-08 16:37:36.144490", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-09-08 16:37:36.139652", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:36.363 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002834", "end": "2020-09-08 16:37:36.335119", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-09-08 16:37:36.332285", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:36.538 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002721", "end": "2020-09-08 16:37:36.518864", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-09-08 16:37:36.516143", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:36.735 INFO:teuthology.task.ansible.out:ok: [smithi120.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015597", "end": "2020-09-08 16:37:36.715604", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-09-08 16:37:36.700007", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-09-08T16:37:36.788 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:36 +0000 (0:00:00.852) 0:02:03.206 ***** 2020-09-08T16:37:36.819 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-09-08T16:37:36.854 INFO:teuthology.task.ansible.out:Tuesday 08 September 2020 16:37:36 +0000 (0:00:00.066) 0:02:03.272 ***** 2020-09-08T16:37:37.061 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-09-08T16:37:37.063 INFO:teuthology.task.ansible.out:changed: [smithi120.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-09-08T16:37:37.167 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-09-08T16:37:37.171 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2020-09-08T16:37:37.172 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-09-08T16:37:37.175 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-09-08T16:37:37.176 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi120.front.sepia.ceph.com : ok=111 changed=31 unreachable=0 failed=0 skipped=178 rescued=0 ignored=3 Tuesday 08 September 2020 16:37:37 +0000 (0:00:00.321) 0:02:03.593 ***** =============================================================================== common : nrpe - Remove SELinux policy package -------------------------- 16.49s common : nrpe - Load SELinux policy package ---------------------------- 16.42s testnode : Zap all non-root disks --------------------------------------- 8.09s testnode : Install packages --------------------------------------------- 3.08s Gathering Facts --------------------------------------------------------- 2.67s testnode : Ensure ceph packages are not present. ------------------------ 2.42s testnode : Create logical volume(s) ------------------------------------- 2.19s testnode : Ensure ceph dependency packages are not present. ------------- 2.05s testnode : Rebuild rpmdb ------------------------------------------------ 2.01s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.99s common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.61s testnode : Configure local mirrorlists ---------------------------------- 1.55s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.54s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.53s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.51s testnode : Install packages via pip ------------------------------------- 1.44s common : Install Kerberos Packages (RedHat) ----------------------------- 1.33s testnode : Set the system's hostname ------------------------------------ 1.17s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 1.07s common : Configure epel repos in /etc/yum.repos.d/ ------ 2020-09-08T16:37:37.178 INFO:teuthology.task.ansible.out:---------------- 1.07s 2020-09-08T16:37:37.589 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2020-09-08T16:37:37.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-09-08T16:37:37.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-09-08T16:37:37.666 INFO:teuthology.orchestra.run.smithi120:> true 2020-09-08T16:37:37.753 INFO:teuthology.run_tasks:Running task clock... 2020-09-08T16:37:37.767 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-09-08T16:37:37.767 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-09-08T16:37:37.768 INFO:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-09-08T16:37:37.827 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-09-08T16:37:37.844 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-09-08T16:37:37.886 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2020-09-08T16:37:37.900 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2020-09-08T16:37:37.917 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-09-08T16:37:37.934 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-09-08T16:37:37.998 INFO:teuthology.orchestra.run.smithi120.stderr:bash: ntpq: command not found 2020-09-08T16:37:38.001 INFO:teuthology.orchestra.run.smithi120.stdout:210 Number of sources = 4 2020-09-08T16:37:38.001 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-09-08T16:37:38.001 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2020-09-08T16:37:38.001 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-09-08T16:37:38.002 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-09-08T16:37:38.002 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-09-08T16:37:38.002 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-09-08T16:37:38.002 INFO:teuthology.run_tasks:Running task install... 2020-09-08T16:37:38.023 DEBUG:teuthology.task.install:project ceph 2020-09-08T16:37:38.023 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'dce4d291bd4243c9802c17d67b85a6da6bbc0f8f'}} 2020-09-08T16:37:38.024 DEBUG:teuthology.task.install:config {'sha1': 'dce4d291bd4243c9802c17d67b85a6da6bbc0f8f'} 2020-09-08T16:37:38.024 INFO:teuthology.task.install:Using flavor: basic 2020-09-08T16:37:38.031 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-09-08T16:37:38.031 INFO:teuthology.task.install:extra packages: [] 2020-09-08T16:37:38.032 INFO:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2020-09-08T16:37:38.077 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS 2020-09-08T16:37:38.078 DEBUG:teuthology.misc:System to be installed: CentOS 2020-09-08T16:37:38.079 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'dce4d291bd4243c9802c17d67b85a6da6bbc0f8f', 'tag': None, 'wait_for_package': False} 2020-09-08T16:37:38.080 INFO:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2020-09-08T16:37:38.153 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS 2020-09-08T16:37:38.154 DEBUG:teuthology.misc:System to be installed: CentOS 2020-09-08T16:37:38.155 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=dce4d291bd4243c9802c17d67b85a6da6bbc0f8f 2020-09-08T16:37:38.338 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-09-08-0934/dce4d291bd4243c9802c17d67b85a6da6bbc0f8f/centos/8/flavors/default/ 2020-09-08T16:37:38.339 INFO:teuthology.task.install.rpm:Package version is 16.0.0-5272.gdce4d291bd4 2020-09-08T16:37:38.644 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-09-08-0934/dce4d291bd4243c9802c17d67b85a6da6bbc0f8f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-09-08-0934/dce4d291bd4243c9802c17d67b85a6da6bbc0f8f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-kefu-testing-2020-09-08-0934/dce4d291bd4243c9802c17d67b85a6da6bbc0f8f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-09-08T16:37:38.645 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-09-08T16:37:38.645 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2020-09-08T16:37:38.659 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/etc/yum.repos.d/ceph.repo 2020-09-08T16:37:38.672 INFO:teuthology.orchestra.run.smithi120.stderr:1+1 records in 2020-09-08T16:37:38.673 INFO:teuthology.orchestra.run.smithi120.stderr:1+1 records out 2020-09-08T16:37:38.673 INFO:teuthology.orchestra.run.smithi120.stderr:664 bytes copied, 3.8467e-05 s, 17.3 MB/s 2020-09-08T16:37:38.674 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-09-08T16:37:38.675 INFO:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/dce4d291bd4243c9802c17d67b85a6da6bbc0f8f/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-09-08T16:37:38.743 INFO:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-09-08T16:37:38.824 INFO:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-09-08T16:37:38.897 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2020-09-08T16:37:38.898 INFO:teuthology.orchestra.run.smithi120:> sudo yum clean all 2020-09-08T16:37:39.174 INFO:teuthology.orchestra.run.smithi120.stdout:41 files removed 2020-09-08T16:37:39.204 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw 2020-09-08T16:37:40.241 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 104 kB/s | 72 kB 00:00 2020-09-08T16:37:40.795 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 28 kB/s | 14 kB 00:00 2020-09-08T16:37:41.368 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 2.6 kB/s | 1.4 kB 00:00 2020-09-08T16:37:41.543 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - AppStream 46 MB/s | 7.0 MB 00:00 2020-09-08T16:37:42.727 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Base 25 MB/s | 2.2 MB 00:00 2020-09-08T16:37:43.158 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-8 - Extras 88 kB/s | 5.5 kB 00:00 2020-09-08T16:37:43.525 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 25 MB/s | 8.0 MB 00:00 2020-09-08T16:37:45.180 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 4.0 MB/s | 21 kB 00:00 2020-09-08T16:37:46.194 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:37:46.207 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:37:46.207 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-09-08T16:37:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:37:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:37:46.208 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 11 M 2020-09-08T16:37:46.208 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2020-09-08T16:37:46.208 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 3.7 M 2020-09-08T16:37:46.209 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 2.2 M 2020-09-08T16:37:46.209 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:37:46.209 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 6.0 M 2020-09-08T16:37:46.209 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 23 M 2020-09-08T16:37:46.209 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 25 k 2020-09-08T16:37:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 553 k 2020-09-08T16:37:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 485 k 2020-09-08T16:37:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 3.4 M 2020-09-08T16:37:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse 2020-09-08T16:37:46.210 INFO:teuthology.orchestra.run.smithi120.stdout: x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 40 k 2020-09-08T16:37:46.211 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common 2020-09-08T16:37:46.211 INFO:teuthology.orchestra.run.smithi120.stdout: x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 72 k 2020-09-08T16:37:46.211 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 204 k 2020-09-08T16:37:46.211 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 364 k 2020-09-08T16:37:46.211 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 335 k 2020-09-08T16:37:46.212 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 113 k 2020-09-08T16:37:46.212 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-09-08T16:37:46.212 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-09-08T16:37:46.212 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable 2020-09-08T16:37:46.212 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-09-08T16:37:46.213 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-09-08T16:37:46.213 INFO:teuthology.orchestra.run.smithi120.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2020-09-08T16:37:46.213 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-09-08T16:37:46.213 INFO:teuthology.orchestra.run.smithi120.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2020-09-08T16:37:46.213 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-09-08T16:37:46.213 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-09-08T16:37:46.214 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:37:46.214 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:37:46.214 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:37:46.214 INFO:teuthology.orchestra.run.smithi120.stdout:Install 22 Packages 2020-09-08T16:37:46.214 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 2 Packages 2020-09-08T16:37:46.215 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:37:46.215 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 52 M 2020-09-08T16:37:46.216 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:37:47.666 INFO:teuthology.orchestra.run.smithi120.stdout:(1/24): ceph-base-16.0.0-5272.gdce4d291bd4.el8. 4.1 MB/s | 6.0 MB 00:01 2020-09-08T16:37:47.707 INFO:teuthology.orchestra.run.smithi120.stdout:(2/24): ceph-selinux-16.0.0-5272.gdce4d291bd4.e 594 kB/s | 25 kB 00:00 2020-09-08T16:37:47.810 INFO:teuthology.orchestra.run.smithi120.stdout:(3/24): libcephfs2-16.0.0-5272.gdce4d291bd4.el8 5.3 MB/s | 553 kB 00:00 2020-09-08T16:37:47.925 INFO:teuthology.orchestra.run.smithi120.stdout:(4/24): libradosstriper1-16.0.0-5272.gdce4d291b 4.1 MB/s | 485 kB 00:00 2020-09-08T16:37:48.256 INFO:teuthology.orchestra.run.smithi120.stdout:(5/24): ceph-radosgw-16.0.0-5272.gdce4d291bd4.e 5.2 MB/s | 11 MB 00:02 2020-09-08T16:37:48.322 INFO:teuthology.orchestra.run.smithi120.stdout:(6/24): python3-ceph-argparse-16.0.0-5272.gdce4 604 kB/s | 40 kB 00:00 2020-09-08T16:37:48.402 INFO:teuthology.orchestra.run.smithi120.stdout:(7/24): python3-ceph-common-16.0.0-5272.gdce4d2 907 kB/s | 72 kB 00:00 2020-09-08T16:37:48.433 INFO:teuthology.orchestra.run.smithi120.stdout:(8/24): librgw2-16.0.0-5272.gdce4d291bd4.el8.x8 6.7 MB/s | 3.4 MB 00:00 2020-09-08T16:37:48.472 INFO:teuthology.orchestra.run.smithi120.stdout:(9/24): python3-cephfs-16.0.0-5272.gdce4d291bd4 2.8 MB/s | 204 kB 00:00 2020-09-08T16:37:48.516 INFO:teuthology.orchestra.run.smithi120.stdout:(10/24): python3-rados-16.0.0-5272.gdce4d291bd4 4.4 MB/s | 364 kB 00:00 2020-09-08T16:37:48.586 INFO:teuthology.orchestra.run.smithi120.stdout:(11/24): python3-rgw-16.0.0-5272.gdce4d291bd4.e 1.5 MB/s | 113 kB 00:00 2020-09-08T16:37:48.621 INFO:teuthology.orchestra.run.smithi120.stdout:(12/24): librdkafka-0.11.4-1.el8.x86_64.rpm 9.9 MB/s | 353 kB 00:00 2020-09-08T16:37:48.630 INFO:teuthology.orchestra.run.smithi120.stdout:(13/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 65 MB/s | 259 kB 00:00 2020-09-08T16:37:48.630 INFO:teuthology.orchestra.run.smithi120.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 32 MB/s | 44 kB 00:00 2020-09-08T16:37:48.631 INFO:teuthology.orchestra.run.smithi120.stdout:(15/24): librabbitmq-0.9.0-1.el8.x86_64.rpm 32 MB/s | 47 kB 00:00 2020-09-08T16:37:48.726 INFO:teuthology.orchestra.run.smithi120.stdout:(16/24): fmt-6.2.1-1.el8.x86_64.rpm 1.2 MB/s | 116 kB 00:00 2020-09-08T16:37:48.767 INFO:teuthology.orchestra.run.smithi120.stdout:(17/24): gperftools-libs-2.7-6.el8.x86_64.rpm 7.7 MB/s | 315 kB 00:00 2020-09-08T16:37:48.787 INFO:teuthology.orchestra.run.smithi120.stdout:(18/24): leveldb-1.22-1.el8.x86_64.rpm 8.5 MB/s | 181 kB 00:00 2020-09-08T16:37:48.807 INFO:teuthology.orchestra.run.smithi120.stdout:(19/24): liboath-2.6.2-3.el8.x86_64.rpm 2.9 MB/s | 59 kB 00:00 2020-09-08T16:37:48.827 INFO:teuthology.orchestra.run.smithi120.stdout:(20/24): libunwind-1.3.1-3.el8.x86_64.rpm 3.6 MB/s | 75 kB 00:00 2020-09-08T16:37:49.373 INFO:teuthology.orchestra.run.smithi120.stdout:(21/24): python3-rbd-16.0.0-5272.gdce4d291bd4.e 371 kB/s | 335 kB 00:00 2020-09-08T16:37:49.609 INFO:teuthology.orchestra.run.smithi120.stdout:(22/24): ceph-common-16.0.0-5272.gdce4d291bd4.e 6.7 MB/s | 23 MB 00:03 2020-09-08T16:37:49.631 INFO:teuthology.orchestra.run.smithi120.stdout:(23/24): librados2-16.0.0-5272.gdce4d291bd4.el8 4.6 MB/s | 3.7 MB 00:00 2020-09-08T16:37:49.795 INFO:teuthology.orchestra.run.smithi120.stdout:(24/24): librbd1-16.0.0-5272.gdce4d291bd4.el8.x 5.1 MB/s | 2.2 MB 00:00 2020-09-08T16:37:49.797 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:37:49.798 INFO:teuthology.orchestra.run.smithi120.stdout:Total 15 MB/s | 52 MB 00:03 2020-09-08T16:37:49.803 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:37:49.860 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:37:49.861 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:37:50.423 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:37:50.424 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:37:51.635 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:37:51.772 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2020-09-08T16:37:51.974 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2020-09-08T16:37:52.155 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-09-08T16:37:52.555 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2020-09-08T16:37:52.756 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 4/26 2020-09-08T16:37:52.835 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 4/26 2020-09-08T16:37:53.076 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:16.0.0-5272.gdce4d291bd4.el8.x86_6 5/26 2020-09-08T16:37:53.230 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 6/26 2020-09-08T16:37:53.485 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 6/26 2020-09-08T16:37:53.730 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 7/26 2020-09-08T16:37:53.802 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 7/26 2020-09-08T16:37:54.013 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 8/26 2020-09-08T16:37:54.146 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2020-09-08T16:37:54.544 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2020-09-08T16:37:54.663 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 10/26 2020-09-08T16:37:54.765 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 10/26 2020-09-08T16:37:54.934 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2020-09-08T16:37:55.092 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:16.0.0-5272.gdce4d291bd4.e 12/26 2020-09-08T16:37:55.268 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:16.0.0-5272.gdce4d291bd4.el8.x86_ 13/26 2020-09-08T16:37:55.436 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 14/26 2020-09-08T16:37:55.648 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 15/26 2020-09-08T16:42:56.849 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:16.0.0-5272.gdce4d291bd4.el8.x8 16/26 2020-09-08T16:42:56.850 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:16.0.0-5272.gdce4d291bd4.el8.x8 16/26 2020-09-08T16:42:56.850 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2020-09-08T16:42:56.851 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 18/26 2020-09-08T16:42:56.851 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2020-09-08T16:42:56.851 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:16.0.0-5272.gdce4d291bd4.el8 20/26 2020-09-08T16:42:56.851 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 21/26 2020-09-08T16:42:56.851 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 21/26 2020-09-08T16:42:56.852 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 21/26 2020-09-08T16:42:56.852 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 22/26 2020-09-08T16:42:56.852 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 22/26 2020-09-08T16:42:56.852 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 23/26 2020-09-08T16:42:56.853 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 23/26 2020-09-08T16:42:56.853 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2020-09-08T16:42:56.853 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2020-09-08T16:42:56.853 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /dev 2020-09-08T16:42:56.853 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /run 2020-09-08T16:42:56.854 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2020-09-08T16:42:56.855 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2020-09-08T16:42:56.855 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2020-09-08T16:42:56.855 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2020-09-08T16:42:56.855 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /dev 2020-09-08T16:42:56.855 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:42:56.856 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 24/26 2020-09-08T16:42:56.856 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 24/26 2020-09-08T16:42:56.856 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-09-08T16:42:56.856 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2020-09-08T16:42:56.857 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2020-09-08T16:42:56.857 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2020-09-08T16:42:56.857 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/26 2020-09-08T16:42:56.857 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2/26 2020-09-08T16:42:56.857 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 3/26 2020-09-08T16:42:56.858 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 4/26 2020-09-08T16:42:56.858 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 5/26 2020-09-08T16:42:56.858 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:16.0.0-5272.gdce4d291bd4.el8.x8 6/26 2020-09-08T16:42:56.858 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 7/26 2020-09-08T16:42:56.858 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:16.0.0-5272.gdce4d291bd4.e 8/26 2020-09-08T16:42:56.859 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:16.0.0-5272.gdce4d291bd4.el8 9/26 2020-09-08T16:42:56.859 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:16.0.0-5272.gdce4d291bd4.el8.x86_ 10/26 2020-09-08T16:42:56.859 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:16.0.0-5272.gdce4d291bd4.el8.x86_6 11/26 2020-09-08T16:42:56.859 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 12/26 2020-09-08T16:42:56.859 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 13/26 2020-09-08T16:42:56.860 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2020-09-08T16:42:56.860 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2020-09-08T16:42:56.860 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2020-09-08T16:42:56.860 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/26 2020-09-08T16:42:56.861 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2020-09-08T16:42:56.861 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 19/26 2020-09-08T16:42:56.861 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2020-09-08T16:42:56.861 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2020-09-08T16:42:56.861 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2020-09-08T16:42:56.862 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 23/26 2020-09-08T16:42:56.862 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2020-09-08T16:42:56.862 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 25/26 2020-09-08T16:42:56.862 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2020-09-08T16:42:56.862 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:42:56.863 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2020-09-08T16:42:56.863 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.863 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.863 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:42:56.864 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:42:56.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.864 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.865 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.865 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.865 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.865 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.865 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.866 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.866 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.866 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:42:56.867 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-09-08T16:42:56.867 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-09-08T16:42:56.867 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-09-08T16:42:56.867 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-09-08T16:42:56.868 INFO:teuthology.orchestra.run.smithi120.stdout: fmt-6.2.1-1.el8.x86_64 2020-09-08T16:42:56.868 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-09-08T16:42:56.868 INFO:teuthology.orchestra.run.smithi120.stdout: leveldb-1.22-1.el8.x86_64 2020-09-08T16:42:56.868 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.2-3.el8.x86_64 2020-09-08T16:42:56.868 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.3.1-3.el8.x86_64 2020-09-08T16:42:56.869 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:42:56.869 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:42:56.870 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-test 2020-09-08T16:42:57.298 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:12 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:42:57.695 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:42:57.696 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:42:57.696 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-09-08T16:42:57.697 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:42:57.697 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:42:57.697 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 46 M 2020-09-08T16:42:57.697 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:42:57.697 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-09-08T16:42:57.698 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-09-08T16:42:57.698 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-09-08T16:42:57.698 INFO:teuthology.orchestra.run.smithi120.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-09-08T16:42:57.698 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-09-08T16:42:57.699 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:42:57.699 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:42:57.699 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:42:57.699 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2020-09-08T16:42:57.699 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:42:57.700 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 47 M 2020-09-08T16:42:57.700 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 200 M 2020-09-08T16:42:57.700 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:42:57.734 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.5 MB/s | 161 kB 00:00 2020-09-08T16:42:57.738 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 4.8 MB/s | 188 kB 00:00 2020-09-08T16:42:57.740 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 48 MB/s | 298 kB 00:00 2020-09-08T16:42:57.742 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 56 MB/s | 249 kB 00:00 2020-09-08T16:42:57.806 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.0 MB/s | 69 kB 00:00 2020-09-08T16:42:59.864 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-test-16.0.0-5272.gdce4d291bd4.el8.x 21 MB/s | 46 MB 00:02 2020-09-08T16:42:59.866 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:42:59.867 INFO:teuthology.orchestra.run.smithi120.stdout:Total 22 MB/s | 47 MB 00:02 2020-09-08T16:42:59.873 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:42:59.896 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:42:59.896 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:00.189 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:00.189 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:00.726 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:00.883 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-09-08T16:43:01.122 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-09-08T16:43:01.292 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-09-08T16:43:01.479 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-09-08T16:43:01.546 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-09-08T16:43:05.936 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-09-08T16:43:06.187 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 6/6 2020-09-08T16:43:06.392 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-test-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 6/6 2020-09-08T16:43:06.393 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/6 2020-09-08T16:43:06.394 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-09-08T16:43:06.394 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-09-08T16:43:06.395 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-09-08T16:43:06.395 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-09-08T16:43:06.744 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-09-08T16:43:06.744 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:06.745 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:06.745 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:06.747 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.5-12.el8.x86_64 2020-09-08T16:43:06.747 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.8.2-1.el8.x86_64 2020-09-08T16:43:06.747 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.3.2-6.el8.x86_64 2020-09-08T16:43:06.747 INFO:teuthology.orchestra.run.smithi120.stdout: libxslt-1.1.32-3.el8.x86_64 2020-09-08T16:43:06.747 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2020-09-08T16:43:06.748 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:06.748 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:07.014 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph 2020-09-08T16:43:07.428 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:22 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:07.795 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:07.799 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:07.800 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-09-08T16:43:07.800 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:07.800 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:07.800 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:43:07.800 INFO:teuthology.orchestra.run.smithi120.stdout: ceph 6.5 k 2020-09-08T16:43:07.801 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:43:07.801 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:43:07.801 INFO:teuthology.orchestra.run.smithi120.stdout: ceph 2.2 M 2020-09-08T16:43:07.801 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:43:07.802 INFO:teuthology.orchestra.run.smithi120.stdout: ceph 1.5 M 2020-09-08T16:43:07.802 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:43:07.802 INFO:teuthology.orchestra.run.smithi120.stdout: ceph 4.9 M 2020-09-08T16:43:07.802 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:43:07.802 INFO:teuthology.orchestra.run.smithi120.stdout: ceph 18 M 2020-09-08T16:43:07.803 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:43:07.803 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-noarch 189 k 2020-09-08T16:43:07.803 INFO:teuthology.orchestra.run.smithi120.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2020-09-08T16:43:07.803 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2020-09-08T16:43:07.803 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2020-09-08T16:43:07.804 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2020-09-08T16:43:07.804 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2020-09-08T16:43:07.804 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2020-09-08T16:43:07.804 INFO:teuthology.orchestra.run.smithi120.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2020-09-08T16:43:07.804 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2020-09-08T16:43:07.805 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-09-08T16:43:07.805 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.24.2-2.el8 CentOS-Base 176 k 2020-09-08T16:43:07.805 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2020-09-08T16:43:07.805 INFO:teuthology.orchestra.run.smithi120.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2020-09-08T16:43:07.805 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.2.1-1.el8 epel 145 k 2020-09-08T16:43:07.806 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2020-09-08T16:43:07.806 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2020-09-08T16:43:07.806 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2020-09-08T16:43:07.806 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2020-09-08T16:43:07.806 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2020-09-08T16:43:07.806 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2020-09-08T16:43:07.807 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2020-09-08T16:43:07.807 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2020-09-08T16:43:07.807 INFO:teuthology.orchestra.run.smithi120.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2020-09-08T16:43:07.807 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2020-09-08T16:43:07.807 INFO:teuthology.orchestra.run.smithi120.stdout: python3-trustme noarch 0.5.2-4.el8 epel 26 k 2020-09-08T16:43:07.808 INFO:teuthology.orchestra.run.smithi120.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2020-09-08T16:43:07.808 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2020-09-08T16:43:07.808 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2020-09-08T16:43:07.808 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2020-09-08T16:43:07.808 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2020-09-08T16:43:07.809 INFO:teuthology.orchestra.run.smithi120.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-09-08T16:43:07.809 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2020-09-08T16:43:07.809 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:07.809 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:07.809 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:07.810 INFO:teuthology.orchestra.run.smithi120.stdout:Install 36 Packages 2020-09-08T16:43:07.810 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:07.810 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 31 M 2020-09-08T16:43:07.811 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 110 M 2020-09-08T16:43:07.811 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:07.906 INFO:teuthology.orchestra.run.smithi120.stdout:(1/36): ceph-16.0.0-5272.gdce4d291bd4.el8.x86_6 64 kB/s | 6.5 kB 00:00 2020-09-08T16:43:08.164 INFO:teuthology.orchestra.run.smithi120.stdout:(2/36): ceph-mds-16.0.0-5272.gdce4d291bd4.el8.x 6.0 MB/s | 2.2 MB 00:00 2020-09-08T16:43:08.172 INFO:teuthology.orchestra.run.smithi120.stdout:(3/36): ceph-mgr-16.0.0-5272.gdce4d291bd4.el8.x 4.2 MB/s | 1.5 MB 00:00 2020-09-08T16:43:08.240 INFO:teuthology.orchestra.run.smithi120.stdout:(4/36): ceph-mgr-modules-core-16.0.0-5272.gdce4 2.9 MB/s | 189 kB 00:00 2020-09-08T16:43:08.268 INFO:teuthology.orchestra.run.smithi120.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.6 MB/s | 214 kB 00:00 2020-09-08T16:43:08.289 INFO:teuthology.orchestra.run.smithi120.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 75 MB/s | 1.5 MB 00:00 2020-09-08T16:43:08.291 INFO:teuthology.orchestra.run.smithi120.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 58 MB/s | 157 kB 00:00 2020-09-08T16:43:08.293 INFO:teuthology.orchestra.run.smithi120.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 27 MB/s | 39 kB 00:00 2020-09-08T16:43:08.295 INFO:teuthology.orchestra.run.smithi120.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2020-09-08T16:43:08.297 INFO:teuthology.orchestra.run.smithi120.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 20 MB/s | 27 kB 00:00 2020-09-08T16:43:08.302 INFO:teuthology.orchestra.run.smithi120.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 76 MB/s | 457 kB 00:00 2020-09-08T16:43:08.306 INFO:teuthology.orchestra.run.smithi120.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 59 MB/s | 195 kB 00:00 2020-09-08T16:43:08.307 INFO:teuthology.orchestra.run.smithi120.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 26 MB/s | 34 kB 00:00 2020-09-08T16:43:08.311 INFO:teuthology.orchestra.run.smithi120.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 54 MB/s | 123 kB 00:00 2020-09-08T16:43:08.313 INFO:teuthology.orchestra.run.smithi120.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 60 MB/s | 176 kB 00:00 2020-09-08T16:43:08.383 INFO:teuthology.orchestra.run.smithi120.stdout:(16/36): ceph-mon-16.0.0-5272.gdce4d291bd4.el8. 10 MB/s | 4.9 MB 00:00 2020-09-08T16:43:08.390 INFO:teuthology.orchestra.run.smithi120.stdout:(17/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 561 kB/s | 44 kB 00:00 2020-09-08T16:43:08.454 INFO:teuthology.orchestra.run.smithi120.stdout:(18/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.4 MB/s | 145 kB 00:00 2020-09-08T16:43:08.464 INFO:teuthology.orchestra.run.smithi120.stdout:(19/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.2 MB/s | 185 kB 00:00 2020-09-08T16:43:08.480 INFO:teuthology.orchestra.run.smithi120.stdout:(20/36): python3-cssselect-0.9.2-10.el8.noarch. 2.4 MB/s | 40 kB 00:00 2020-09-08T16:43:08.489 INFO:teuthology.orchestra.run.smithi120.stdout:(21/36): python3-cherrypy-18.4.0-1.el8.noarch.r 9.3 MB/s | 384 kB 00:00 2020-09-08T16:43:08.497 INFO:teuthology.orchestra.run.smithi120.stdout:(22/36): python3-jaraco-6.2-6.el8.noarch.rpm 683 kB/s | 11 kB 00:00 2020-09-08T16:43:08.510 INFO:teuthology.orchestra.run.smithi120.stdout:(23/36): python3-jaraco-functools-2.0-4.el8.noa 891 kB/s | 18 kB 00:00 2020-09-08T16:43:08.513 INFO:teuthology.orchestra.run.smithi120.stdout:(24/36): python3-logutils-0.3.5-11.el8.noarch.r 2.9 MB/s | 49 kB 00:00 2020-09-08T16:43:08.529 INFO:teuthology.orchestra.run.smithi120.stdout:(25/36): python3-more-itertools-7.2.0-3.el8.noa 2.9 MB/s | 59 kB 00:00 2020-09-08T16:43:08.546 INFO:teuthology.orchestra.run.smithi120.stdout:(26/36): python3-pecan-1.3.2-9.el8.noarch.rpm 8.4 MB/s | 283 kB 00:00 2020-09-08T16:43:08.549 INFO:teuthology.orchestra.run.smithi120.stdout:(27/36): python3-portend-2.6-1.el8.noarch.rpm 788 kB/s | 16 kB 00:00 2020-09-08T16:43:08.563 INFO:teuthology.orchestra.run.smithi120.stdout:(28/36): python3-simplegeneric-0.8.1-17.el8.noa 1.1 MB/s | 19 kB 00:00 2020-09-08T16:43:08.569 INFO:teuthology.orchestra.run.smithi120.stdout:(29/36): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2020-09-08T16:43:08.579 INFO:teuthology.orchestra.run.smithi120.stdout:(30/36): python3-tempora-1.14.1-5.el8.noarch.rp 1.7 MB/s | 29 kB 00:00 2020-09-08T16:43:08.589 INFO:teuthology.orchestra.run.smithi120.stdout:(31/36): python3-trustme-0.5.2-4.el8.noarch.rpm 1.3 MB/s | 26 kB 00:00 2020-09-08T16:43:08.598 INFO:teuthology.orchestra.run.smithi120.stdout:(32/36): python3-waitress-1.2.1-2.el8.1.noarch. 13 MB/s | 241 kB 00:00 2020-09-08T16:43:08.613 INFO:teuthology.orchestra.run.smithi120.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 11 MB/s | 251 kB 00:00 2020-09-08T16:43:08.614 INFO:teuthology.orchestra.run.smithi120.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 5.0 MB/s | 85 kB 00:00 2020-09-08T16:43:08.630 INFO:teuthology.orchestra.run.smithi120.stdout:(35/36): python3-zc-lockfile-2.0-2.el8.noarch.r 1.2 MB/s | 23 kB 00:00 2020-09-08T16:43:09.019 INFO:teuthology.orchestra.run.smithi120.stdout:(36/36): ceph-osd-16.0.0-5272.gdce4d291bd4.el8. 21 MB/s | 18 MB 00:00 2020-09-08T16:43:09.020 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:09.020 INFO:teuthology.orchestra.run.smithi120.stdout:Total 26 MB/s | 31 MB 00:01 2020-09-08T16:43:09.029 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:09.068 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:09.069 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:09.372 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:09.373 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:10.752 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:10.973 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/36 2020-09-08T16:43:11.237 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/36 2020-09-08T16:43:11.432 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/36 2020-09-08T16:43:11.637 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/36 2020-09-08T16:43:11.794 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 5/36 2020-09-08T16:43:11.992 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/36 2020-09-08T16:43:12.140 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/36 2020-09-08T16:43:12.316 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/36 2020-09-08T16:43:12.444 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/36 2020-09-08T16:43:12.686 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/36 2020-09-08T16:43:12.966 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 11/36 2020-09-08T16:43:13.157 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 12/36 2020-09-08T16:43:13.370 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 13/36 2020-09-08T16:43:13.559 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-09-08T16:43:13.849 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-09-08T16:43:14.094 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-09-08T16:43:14.359 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-09-08T16:43:14.599 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-09-08T16:43:14.833 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-09-08T16:43:15.066 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-09-08T16:43:15.439 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-09-08T16:43:15.699 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-09-08T16:43:15.893 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-09-08T16:43:16.065 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-09-08T16:43:16.201 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-09-08T16:43:16.438 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-09-08T16:43:16.658 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-09-08T16:43:16.915 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-09-08T16:43:17.267 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-09-08T16:43:17.618 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-09-08T16:43:17.976 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-5272.gdce4d291bd4.e 31/36 2020-09-08T16:43:18.116 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 32/36 2020-09-08T16:43:19.632 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 32/36 2020-09-08T16:43:19.829 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 33/36 2020-09-08T16:43:20.255 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 33/36 2020-09-08T16:43:20.428 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 34/36 2020-09-08T16:43:20.630 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 34/36 2020-09-08T16:43:20.803 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 35/36 2020-09-08T16:43:20.821 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 35/36 2020-09-08T16:43:20.910 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 36/36 2020-09-08T16:43:21.202 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 36/36 2020-09-08T16:43:21.202 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/36 2020-09-08T16:43:21.202 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2/36 2020-09-08T16:43:21.203 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 3/36 2020-09-08T16:43:21.203 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 4/36 2020-09-08T16:43:21.203 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 5/36 2020-09-08T16:43:21.203 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-5272.gdce4d291bd4.e 6/36 2020-09-08T16:43:21.204 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-09-08T16:43:21.204 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-09-08T16:43:21.204 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-09-08T16:43:21.204 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-09-08T16:43:21.204 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-09-08T16:43:21.204 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-09-08T16:43:21.205 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-09-08T16:43:21.205 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-09-08T16:43:21.205 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-09-08T16:43:21.205 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-09-08T16:43:21.205 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-09-08T16:43:21.206 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/36 2020-09-08T16:43:21.206 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/36 2020-09-08T16:43:21.206 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 20/36 2020-09-08T16:43:21.206 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 21/36 2020-09-08T16:43:21.206 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 22/36 2020-09-08T16:43:21.207 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 23/36 2020-09-08T16:43:21.207 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 24/36 2020-09-08T16:43:21.207 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 25/36 2020-09-08T16:43:21.207 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 26/36 2020-09-08T16:43:21.207 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 27/36 2020-09-08T16:43:21.208 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-2.6-1.el8.noarch 28/36 2020-09-08T16:43:21.209 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 29/36 2020-09-08T16:43:21.209 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 30/36 2020-09-08T16:43:21.209 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 31/36 2020-09-08T16:43:21.209 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 32/36 2020-09-08T16:43:21.210 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 33/36 2020-09-08T16:43:21.210 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 34/36 2020-09-08T16:43:21.210 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 35/36 2020-09-08T16:43:22.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 36/36 2020-09-08T16:43:22.591 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:22.592 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:22.592 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:22.592 INFO:teuthology.orchestra.run.smithi120.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-09-08T16:43:22.592 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-09-08T16:43:22.592 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:22.593 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:22.593 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:22.593 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:22.593 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:16.0.0-5272.gdce4d291bd4.el8.noarch 2020-09-08T16:43:22.593 INFO:teuthology.orchestra.run.smithi120.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-09-08T16:43:22.594 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.0.6-13.el8.noarch 2020-09-08T16:43:22.594 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-09-08T16:43:22.594 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2017.2-9.el8.noarch 2020-09-08T16:43:22.594 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-09-08T16:43:22.594 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-09-08T16:43:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-09-08T16:43:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-09-08T16:43:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-09-08T16:43:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-09-08T16:43:22.595 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-09-08T16:43:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-09-08T16:43:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-09-08T16:43:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-09-08T16:43:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-6.2-6.el8.noarch 2020-09-08T16:43:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-09-08T16:43:22.596 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-09-08T16:43:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-09-08T16:43:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-09-08T16:43:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-2.6-1.el8.noarch 2020-09-08T16:43:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-09-08T16:43:22.597 INFO:teuthology.orchestra.run.smithi120.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-09-08T16:43:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-09-08T16:43:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-09-08T16:43:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-09-08T16:43:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-09-08T16:43:22.598 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-09-08T16:43:22.599 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-09-08T16:43:22.599 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:22.599 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:22.885 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-base 2020-09-08T16:43:23.304 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:38 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:23.578 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-base-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:43:23.688 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:23.689 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:43:23.689 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:23.793 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install cephadm 2020-09-08T16:43:24.199 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:39 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:24.564 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:24.565 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:24.566 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-09-08T16:43:24.566 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:24.566 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:24.566 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 55 k 2020-09-08T16:43:24.566 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:24.567 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:24.567 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:24.567 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-09-08T16:43:24.567 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:24.567 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 55 k 2020-09-08T16:43:24.568 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 206 k 2020-09-08T16:43:24.568 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:24.712 INFO:teuthology.orchestra.run.smithi120.stdout:cephadm-16.0.0-5272.gdce4d291bd4.el8.x86_64.rpm 376 kB/s | 55 kB 00:00 2020-09-08T16:43:24.713 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:24.713 INFO:teuthology.orchestra.run.smithi120.stdout:Total 373 kB/s | 55 kB 00:00 2020-09-08T16:43:24.715 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:24.722 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:24.722 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:24.724 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:24.724 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:24.769 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:26.273 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:43:26.393 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:43:26.536 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:43:26.794 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:43:26.794 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:26.794 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:26.795 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:26.795 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:26.795 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:27.090 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-immutable-object-cache 2020-09-08T16:43:27.509 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:42 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:27.949 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:27.950 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:27.950 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-09-08T16:43:27.950 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:27.951 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:27.951 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 203 k 2020-09-08T16:43:27.951 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:27.951 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:27.951 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:27.952 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-09-08T16:43:27.952 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:27.952 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 203 k 2020-09-08T16:43:27.952 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 692 k 2020-09-08T16:43:27.952 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:28.143 INFO:teuthology.orchestra.run.smithi120.stdout:ceph-immutable-object-cache-16.0.0-5272.gdce4d2 1.0 MB/s | 203 kB 00:00 2020-09-08T16:43:28.144 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:28.144 INFO:teuthology.orchestra.run.smithi120.stdout:Total 1.0 MB/s | 203 kB 00:00 2020-09-08T16:43:28.145 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:28.158 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:28.158 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:28.206 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:28.206 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:28.318 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:28.485 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-5272.gdce4d291b 1/1 2020-09-08T16:43:28.799 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-5272.gdce4d291b 1/1 2020-09-08T16:43:29.062 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-5272.gdce4d291b 1/1 2020-09-08T16:43:29.063 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:29.063 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:29.063 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:43:29.063 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:29.064 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:29.315 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr 2020-09-08T16:43:29.729 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:44 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:30.042 INFO:teuthology.orchestra.run.smithi120.stdout:Package ceph-mgr-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:43:30.153 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:30.154 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:43:30.155 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:30.245 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-dashboard 2020-09-08T16:43:30.653 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:45 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:31.014 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:31.014 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:31.015 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-09-08T16:43:31.015 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:31.015 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:31.015 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:16.0.0-5272.gdce4d291bd4.el8 ceph-noarch 3.2 M 2020-09-08T16:43:31.015 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:43:31.016 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards 2020-09-08T16:43:31.016 INFO:teuthology.orchestra.run.smithi120.stdout: noarch 2:16.0.0-5272.gdce4d291bd4.el8 ceph-noarch 24 k 2020-09-08T16:43:31.016 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:16.0.0-5272.gdce4d291bd4.el8 ceph-noarch 8.8 k 2020-09-08T16:43:31.016 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2020-09-08T16:43:31.016 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2020-09-08T16:43:31.017 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2020-09-08T16:43:31.017 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:31.017 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:31.017 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:31.017 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2020-09-08T16:43:31.018 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:31.018 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 3.5 M 2020-09-08T16:43:31.019 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 14 M 2020-09-08T16:43:31.019 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:31.126 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): ceph-prometheus-alerts-16.0.0-5272.gdce4 81 kB/s | 8.8 kB 00:00 2020-09-08T16:43:31.139 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): ceph-grafana-dashboards-16.0.0-5272.gdce 195 kB/s | 24 kB 00:00 2020-09-08T16:43:31.158 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2020-09-08T16:43:31.199 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 562 kB/s | 34 kB 00:00 2020-09-08T16:43:31.239 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.4 MB/s | 196 kB 00:00 2020-09-08T16:43:31.338 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): ceph-mgr-dashboard-16.0.0-5272.gdce4d291 9.8 MB/s | 3.2 MB 00:00 2020-09-08T16:43:31.339 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:31.340 INFO:teuthology.orchestra.run.smithi120.stdout:Total 11 MB/s | 3.5 MB 00:00 2020-09-08T16:43:31.343 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:31.352 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:31.352 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:31.513 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:31.516 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:31.788 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:31.974 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2020-09-08T16:43:32.194 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2020-09-08T16:43:32.392 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2020-09-08T16:43:33.317 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-5272.gdce4d291bd4.el 4/6 2020-09-08T16:43:34.355 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-5272.gdce4d291bd4.e 5/6 2020-09-08T16:43:34.638 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-5272.gdce4d291bd4.el8.no 6/6 2020-09-08T16:43:34.844 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-5272.gdce4d291bd4.el8.no 6/6 2020-09-08T16:43:34.845 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-5272.gdce4d291bd4.e 1/6 2020-09-08T16:43:34.845 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-5272.gdce4d291bd4.el8.no 2/6 2020-09-08T16:43:34.845 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-5272.gdce4d291bd4.el 3/6 2020-09-08T16:43:34.846 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2020-09-08T16:43:34.846 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2020-09-08T16:43:35.142 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2020-09-08T16:43:35.143 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:35.143 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:35.143 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:16.0.0-5272.gdce4d291bd4.el8.noarch 2020-09-08T16:43:35.143 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:16.0.0-5272.gdce4d291bd4.el8.noarch 2020-09-08T16:43:35.143 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:16.0.0-5272.gdce4d291bd4.el8.noarch 2020-09-08T16:43:35.144 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-09-08T16:43:35.144 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-09-08T16:43:35.144 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.4.1-12.el8.noarch 2020-09-08T16:43:35.144 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:35.144 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:35.242 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-diskprediction-local 2020-09-08T16:43:35.651 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:05:50 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:36.055 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:36.057 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:36.058 INFO:teuthology.orchestra.run.smithi120.stdout: Package ArchVersion Repository Size 2020-09-08T16:43:36.058 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:36.058 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:36.058 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch2:16.0.0-5272.gdce4d291bd4.el8 ceph-noarch 8.6 M 2020-09-08T16:43:36.059 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:43:36.059 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-09-08T16:43:36.059 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-09-08T16:43:36.059 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-09-08T16:43:36.059 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-09-08T16:43:36.060 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-09-08T16:43:36.060 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-09-08T16:43:36.060 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-09-08T16:43:36.060 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:36.060 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:36.061 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:36.061 INFO:teuthology.orchestra.run.smithi120.stdout:Install 8 Packages 2020-09-08T16:43:36.061 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:36.061 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 37 M 2020-09-08T16:43:36.062 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 205 M 2020-09-08T16:43:36.062 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:36.166 INFO:teuthology.orchestra.run.smithi120.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 40 MB/s | 4.3 MB 00:00 2020-09-08T16:43:36.212 INFO:teuthology.orchestra.run.smithi120.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2020-09-08T16:43:36.219 INFO:teuthology.orchestra.run.smithi120.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 27 MB/s | 225 kB 00:00 2020-09-08T16:43:36.238 INFO:teuthology.orchestra.run.smithi120.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 52 MB/s | 3.7 MB 00:00 2020-09-08T16:43:36.252 INFO:teuthology.orchestra.run.smithi120.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 44 MB/s | 638 kB 00:00 2020-09-08T16:43:36.258 INFO:teuthology.orchestra.run.smithi120.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 30 MB/s | 169 kB 00:00 2020-09-08T16:43:36.471 INFO:teuthology.orchestra.run.smithi120.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 58 MB/s | 15 MB 00:00 2020-09-08T16:43:36.586 INFO:teuthology.orchestra.run.smithi120.stdout:(8/8): ceph-mgr-diskprediction-local-16.0.0-527 16 MB/s | 8.6 MB 00:00 2020-09-08T16:43:36.588 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:36.588 INFO:teuthology.orchestra.run.smithi120.stdout:Total 69 MB/s | 37 MB 00:00 2020-09-08T16:43:36.598 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:36.617 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:36.617 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:36.912 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:36.912 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:37.492 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:37.626 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-09-08T16:43:37.764 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-09-08T16:43:37.997 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-09-08T16:43:38.541 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-09-08T16:43:39.857 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-09-08T16:43:40.407 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-09-08T16:43:40.542 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-09-08T16:43:41.246 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-09-08T16:43:41.580 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-09-08T16:43:41.783 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-09-08T16:43:43.730 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-09-08T16:43:45.056 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-09-08T16:43:45.353 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-5272.gdce4d29 8/8 2020-09-08T16:43:45.828 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-5272.gdce4d29 8/8 2020-09-08T16:43:45.828 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-5272.gdce4d29 1/8 2020-09-08T16:43:45.828 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-09-08T16:43:45.829 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-09-08T16:43:45.829 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-09-08T16:43:45.829 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-09-08T16:43:45.829 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-09-08T16:43:45.829 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-09-08T16:43:46.141 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-09-08T16:43:46.141 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:46.142 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:46.142 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:16.0.0-5272.gdce4d291bd4.el8.noarch 2020-09-08T16:43:46.142 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.3-2.el8.x86_64 2020-09-08T16:43:46.142 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-09-08T16:43:46.142 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-09-08T16:43:46.143 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-09-08T16:43:46.143 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-09-08T16:43:46.143 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-09-08T16:43:46.143 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-09-08T16:43:46.143 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:46.144 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:46.291 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-rook 2020-09-08T16:43:46.702 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:01 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:47.084 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:47.086 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:47.086 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-09-08T16:43:47.086 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:47.087 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:47.087 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:43:47.087 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-noarch 27 k 2020-09-08T16:43:47.087 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:43:47.087 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-09-08T16:43:47.088 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-09-08T16:43:47.088 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-09-08T16:43:47.088 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-09-08T16:43:47.088 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-09-08T16:43:47.088 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-09-08T16:43:47.089 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-09-08T16:43:47.089 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-09-08T16:43:47.089 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-09-08T16:43:47.089 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 11.0.0-2.el8 epel 1.2 M 2020-09-08T16:43:47.089 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 3.4.2-15.el8 epel 73 k 2020-09-08T16:43:47.090 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-09-08T16:43:47.090 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:47.090 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:47.090 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:47.090 INFO:teuthology.orchestra.run.smithi120.stdout:Install 13 Packages 2020-09-08T16:43:47.091 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:47.091 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 1.9 M 2020-09-08T16:43:47.092 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 24 M 2020-09-08T16:43:47.092 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:47.125 INFO:teuthology.orchestra.run.smithi120.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 745 kB/s | 27 kB 00:00 2020-09-08T16:43:47.128 INFO:teuthology.orchestra.run.smithi120.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 510 kB/s | 20 kB 00:00 2020-09-08T16:43:47.132 INFO:teuthology.orchestra.run.smithi120.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 19 MB/s | 126 kB 00:00 2020-09-08T16:43:47.133 INFO:teuthology.orchestra.run.smithi120.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 26 MB/s | 110 kB 00:00 2020-09-08T16:43:47.134 INFO:teuthology.orchestra.run.smithi120.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 57 MB/s | 155 kB 00:00 2020-09-08T16:43:47.135 INFO:teuthology.orchestra.run.smithi120.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 20 MB/s | 43 kB 00:00 2020-09-08T16:43:47.195 INFO:teuthology.orchestra.run.smithi120.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 273 kB/s | 16 kB 00:00 2020-09-08T16:43:47.195 INFO:teuthology.orchestra.run.smithi120.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 545 kB/s | 33 kB 00:00 2020-09-08T16:43:47.212 INFO:teuthology.orchestra.run.smithi120.stdout:(9/13): ceph-mgr-rook-16.0.0-5272.gdce4d291bd4. 220 kB/s | 27 kB 00:00 2020-09-08T16:43:47.234 INFO:teuthology.orchestra.run.smithi120.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 2.0 MB/s | 81 kB 00:00 2020-09-08T16:43:47.254 INFO:teuthology.orchestra.run.smithi120.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.0 MB/s | 61 kB 00:00 2020-09-08T16:43:47.277 INFO:teuthology.orchestra.run.smithi120.stdout:(12/13): python3-rsa-3.4.2-15.el8.noarch.rpm 1.1 MB/s | 73 kB 00:00 2020-09-08T16:43:47.314 INFO:teuthology.orchestra.run.smithi120.stdout:(13/13): python3-kubernetes-11.0.0-2.el8.noarch 9.8 MB/s | 1.2 MB 00:00 2020-09-08T16:43:47.316 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:47.316 INFO:teuthology.orchestra.run.smithi120.stdout:Total 8.4 MB/s | 1.9 MB 00:00 2020-09-08T16:43:47.322 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:47.336 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:47.337 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:47.420 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:47.420 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:47.935 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:48.235 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-09-08T16:43:48.488 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-09-08T16:43:48.701 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-3.4.2-15.el8.noarch 3/13 2020-09-08T16:43:48.914 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-09-08T16:43:49.071 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-09-08T16:43:49.335 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-09-08T16:43:49.612 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-09-08T16:43:49.841 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-09-08T16:43:50.880 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-09-08T16:43:52.064 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 10/13 2020-09-08T16:43:52.239 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-09-08T16:43:52.398 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-09-08T16:43:52.565 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:16.0.0-5272.gdce4d291bd4.el8.noarc 13/13 2020-09-08T16:43:52.768 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-5272.gdce4d291bd4.el8.noarc 13/13 2020-09-08T16:43:52.768 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:16.0.0-5272.gdce4d291bd4.el8.noarc 1/13 2020-09-08T16:43:52.768 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-09-08T16:43:52.769 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-09-08T16:43:52.769 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-09-08T16:43:52.769 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-09-08T16:43:52.769 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-09-08T16:43:52.769 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-09-08T16:43:52.770 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-09-08T16:43:52.770 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-09-08T16:43:52.770 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-09-08T16:43:52.770 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 11/13 2020-09-08T16:43:52.770 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-3.4.2-15.el8.noarch 12/13 2020-09-08T16:43:53.090 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-09-08T16:43:53.090 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:53.090 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:53.090 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:16.0.0-5272.gdce4d291bd4.el8.noarch 2020-09-08T16:43:53.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-09-08T16:43:53.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-09-08T16:43:53.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-09-08T16:43:53.091 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-09-08T16:43:53.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-09-08T16:43:53.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-09-08T16:43:53.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-09-08T16:43:53.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-09-08T16:43:53.092 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-09-08T16:43:53.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-09-08T16:43:53.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-3.4.2-15.el8.noarch 2020-09-08T16:43:53.093 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-09-08T16:43:53.093 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:53.094 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:53.200 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-mgr-cephadm 2020-09-08T16:43:53.609 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:08 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:53.994 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:53.995 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:53.995 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-09-08T16:43:53.995 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:53.995 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:53.995 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:16.0.0-5272.gdce4d291bd4.el8 ceph-noarch 63 k 2020-09-08T16:43:53.996 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:43:53.996 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2020-09-08T16:43:53.996 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2020-09-08T16:43:53.996 INFO:teuthology.orchestra.run.smithi120.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2020-09-08T16:43:53.996 INFO:teuthology.orchestra.run.smithi120.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2020-09-08T16:43:53.997 INFO:teuthology.orchestra.run.smithi120.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2020-09-08T16:43:53.997 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:53.997 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:53.997 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:53.997 INFO:teuthology.orchestra.run.smithi120.stdout:Install 6 Packages 2020-09-08T16:43:53.998 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:53.998 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 5.8 M 2020-09-08T16:43:53.999 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 24 M 2020-09-08T16:43:53.999 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:54.048 INFO:teuthology.orchestra.run.smithi120.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 10 MB/s | 538 kB 00:00 2020-09-08T16:43:54.090 INFO:teuthology.orchestra.run.smithi120.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 51 MB/s | 4.8 MB 00:00 2020-09-08T16:43:54.118 INFO:teuthology.orchestra.run.smithi120.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 305 kB/s | 21 kB 00:00 2020-09-08T16:43:54.142 INFO:teuthology.orchestra.run.smithi120.stdout:(4/6): ceph-mgr-cephadm-16.0.0-5272.gdce4d291bd 436 kB/s | 63 kB 00:00 2020-09-08T16:43:54.150 INFO:teuthology.orchestra.run.smithi120.stdout:(5/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.4 MB/s | 48 kB 00:00 2020-09-08T16:43:54.189 INFO:teuthology.orchestra.run.smithi120.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.5 MB/s | 355 kB 00:00 2020-09-08T16:43:54.190 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:54.191 INFO:teuthology.orchestra.run.smithi120.stdout:Total 30 MB/s | 5.8 MB 00:00 2020-09-08T16:43:54.195 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:54.204 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:54.204 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:54.293 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:54.294 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:54.548 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:54.847 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2020-09-08T16:43:55.075 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2020-09-08T16:43:55.977 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2020-09-08T16:43:56.319 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2020-09-08T16:43:56.611 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2020-09-08T16:43:56.756 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-5272.gdce4d291bd4.el8.noar 6/6 2020-09-08T16:43:56.947 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-5272.gdce4d291bd4.el8.noar 6/6 2020-09-08T16:43:56.947 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-5272.gdce4d291bd4.el8.noar 1/6 2020-09-08T16:43:56.947 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2020-09-08T16:43:56.947 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2020-09-08T16:43:56.948 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2020-09-08T16:43:56.948 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2020-09-08T16:43:57.790 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2020-09-08T16:43:57.791 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:57.791 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:43:57.791 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:16.0.0-5272.gdce4d291bd4.el8.noarch 2020-09-08T16:43:57.791 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.5.1-5.el8.noarch 2020-09-08T16:43:57.791 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2020-09-08T16:43:57.792 INFO:teuthology.orchestra.run.smithi120.stdout: python3-apipkg-1.5-6.el8.noarch 2020-09-08T16:43:57.792 INFO:teuthology.orchestra.run.smithi120.stdout: python3-execnet-1.7.1-1.el8.noarch 2020-09-08T16:43:57.792 INFO:teuthology.orchestra.run.smithi120.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-09-08T16:43:57.792 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:57.793 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:43:57.902 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-fuse 2020-09-08T16:43:58.322 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:13 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:43:58.721 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:43:58.722 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:58.722 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2020-09-08T16:43:58.722 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:58.723 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:43:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 623 k 2020-09-08T16:43:58.723 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2020-09-08T16:43:58.723 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-09-08T16:43:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-09-08T16:43:58.724 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:58.724 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:43:58.724 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:43:58.724 INFO:teuthology.orchestra.run.smithi120.stdout:Install 3 Packages 2020-09-08T16:43:58.725 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:58.725 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 726 k 2020-09-08T16:43:58.725 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 2.2 M 2020-09-08T16:43:58.725 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:43:58.756 INFO:teuthology.orchestra.run.smithi120.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2020-09-08T16:43:58.757 INFO:teuthology.orchestra.run.smithi120.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 635 kB/s | 21 kB 00:00 2020-09-08T16:43:58.939 INFO:teuthology.orchestra.run.smithi120.stdout:(3/3): ceph-fuse-16.0.0-5272.gdce4d291bd4.el8.x 2.8 MB/s | 623 kB 00:00 2020-09-08T16:43:58.940 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:43:58.941 INFO:teuthology.orchestra.run.smithi120.stdout:Total 3.2 MB/s | 726 kB 00:00 2020-09-08T16:43:58.945 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:43:58.966 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:43:58.966 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:43:59.037 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:43:59.038 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:43:59.233 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:43:59.233 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-09-08T16:43:59.241 INFO:teuthology.orchestra.run.smithi120.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-09-08T16:43:59.241 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:43:59.674 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-09-08T16:43:59.820 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 3/3 2020-09-08T16:44:00.148 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 3/3 2020-09-08T16:44:00.149 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/3 2020-09-08T16:44:00.149 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-09-08T16:44:00.451 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-09-08T16:44:00.451 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:00.452 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:44:00.452 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-09-08T16:44:00.452 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-09-08T16:44:00.452 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:00.452 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:00.782 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install librados-devel 2020-09-08T16:44:01.200 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:16 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:01.657 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:01.658 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:01.658 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-09-08T16:44:01.659 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:01.659 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:44:01.659 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 131 k 2020-09-08T16:44:01.659 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:01.660 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:44:01.660 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:01.660 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-09-08T16:44:01.660 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:01.660 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 131 k 2020-09-08T16:44:01.661 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 436 k 2020-09-08T16:44:01.661 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:44:01.828 INFO:teuthology.orchestra.run.smithi120.stdout:librados-devel-16.0.0-5272.gdce4d291bd4.el8.x86 774 kB/s | 131 kB 00:00 2020-09-08T16:44:01.829 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:44:01.829 INFO:teuthology.orchestra.run.smithi120.stdout:Total 768 kB/s | 131 kB 00:00 2020-09-08T16:44:01.830 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:44:01.837 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:44:01.838 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:44:01.886 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:44:01.887 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:44:01.994 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:44:02.195 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:02.450 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:02.755 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:02.755 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:02.755 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:44:02.756 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:44:02.756 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:02.756 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:03.032 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs2 2020-09-08T16:44:03.450 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:18 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:03.756 INFO:teuthology.orchestra.run.smithi120.stdout:Package libcephfs2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:44:03.871 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:03.872 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:44:03.872 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:03.982 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install libcephfs-devel 2020-09-08T16:44:04.393 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:19 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:04.794 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:04.795 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:04.795 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-09-08T16:44:04.796 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:04.796 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:44:04.796 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 23 k 2020-09-08T16:44:04.796 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:04.797 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:44:04.797 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:04.797 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-09-08T16:44:04.797 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:04.797 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 23 k 2020-09-08T16:44:04.798 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 84 k 2020-09-08T16:44:04.798 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:44:04.919 INFO:teuthology.orchestra.run.smithi120.stdout:libcephfs-devel-16.0.0-5272.gdce4d291bd4.el8.x8 190 kB/s | 23 kB 00:00 2020-09-08T16:44:04.920 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:44:04.920 INFO:teuthology.orchestra.run.smithi120.stdout:Total 188 kB/s | 23 kB 00:00 2020-09-08T16:44:04.921 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:44:04.925 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:44:04.925 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:44:04.927 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:44:04.928 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:44:04.987 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:44:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_6 1/1 2020-09-08T16:44:05.318 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_6 1/1 2020-09-08T16:44:05.546 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_6 1/1 2020-09-08T16:44:05.546 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:05.546 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:44:05.547 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:44:05.547 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:05.547 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:05.648 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install librados2 2020-09-08T16:44:06.058 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:21 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:06.313 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:44:06.426 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:06.427 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:44:06.427 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:06.536 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install librbd1 2020-09-08T16:44:06.949 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:21 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:07.205 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:44:07.319 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:07.320 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:44:07.321 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:07.439 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rados 2020-09-08T16:44:07.851 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:22 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:08.105 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rados-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:44:08.217 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:08.218 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:44:08.218 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:08.345 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rgw 2020-09-08T16:44:08.756 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:23 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:09.013 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rgw-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:44:09.124 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:09.125 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:44:09.126 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:09.228 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-cephfs 2020-09-08T16:44:09.640 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:24 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:09.904 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-cephfs-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:44:10.016 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:10.018 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:44:10.018 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:10.146 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install python3-rbd 2020-09-08T16:44:10.558 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:25 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:10.864 INFO:teuthology.orchestra.run.smithi120.stdout:Package python3-rbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 is already installed. 2020-09-08T16:44:10.977 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:10.978 INFO:teuthology.orchestra.run.smithi120.stdout:Nothing to do. 2020-09-08T16:44:10.978 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:11.078 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-fuse 2020-09-08T16:44:11.488 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:26 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:11.850 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:11.851 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:11.851 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-09-08T16:44:11.852 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:11.852 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:44:11.852 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 87 k 2020-09-08T16:44:11.852 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:11.852 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:44:11.853 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:11.853 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-09-08T16:44:11.853 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:11.853 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 87 k 2020-09-08T16:44:11.853 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 296 k 2020-09-08T16:44:11.854 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:44:11.998 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-fuse-16.0.0-5272.gdce4d291bd4.el8.x86_64.rp 598 kB/s | 87 kB 00:00 2020-09-08T16:44:11.999 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:44:11.999 INFO:teuthology.orchestra.run.smithi120.stdout:Total 594 kB/s | 87 kB 00:00 2020-09-08T16:44:12.001 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:44:12.012 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:44:12.012 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:44:12.061 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:44:12.061 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:44:12.162 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:44:12.326 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:12.541 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:12.803 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:12.803 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:12.803 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:44:12.803 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:44:12.804 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:12.804 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:14.297 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-mirror 2020-09-08T16:44:14.717 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:29 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:15.653 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:15.655 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:15.655 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-09-08T16:44:15.655 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:15.655 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:44:15.655 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 3.1 M 2020-09-08T16:44:15.656 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:15.656 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:44:15.656 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:15.656 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-09-08T16:44:15.656 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:15.657 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 3.1 M 2020-09-08T16:44:15.657 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 14 M 2020-09-08T16:44:15.657 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:44:15.968 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-mirror-16.0.0-5272.gdce4d291bd4.el8.x86_64. 9.9 MB/s | 3.1 MB 00:00 2020-09-08T16:44:15.969 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:44:15.969 INFO:teuthology.orchestra.run.smithi120.stdout:Total 9.8 MB/s | 3.1 MB 00:00 2020-09-08T16:44:15.971 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:44:15.985 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:44:15.986 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:44:16.046 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:44:16.046 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:44:16.497 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:44:16.678 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:16.988 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:17.242 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:17.242 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:17.243 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:44:17.243 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:44:17.243 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:17.243 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:17.559 INFO:teuthology.orchestra.run.smithi120:> sudo yum -y install rbd-nbd 2020-09-08T16:44:17.979 INFO:teuthology.orchestra.run.smithi120.stdout:Last metadata expiration check: 0:06:32 ago on Tue 08 Sep 2020 04:37:45 PM UTC. 2020-09-08T16:44:18.388 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2020-09-08T16:44:18.389 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:18.389 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repo Size 2020-09-08T16:44:18.389 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:18.389 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2020-09-08T16:44:18.390 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:16.0.0-5272.gdce4d291bd4.el8 ceph 161 k 2020-09-08T16:44:18.390 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:18.390 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2020-09-08T16:44:18.390 INFO:teuthology.orchestra.run.smithi120.stdout:================================================================================ 2020-09-08T16:44:18.391 INFO:teuthology.orchestra.run.smithi120.stdout:Install 1 Package 2020-09-08T16:44:18.391 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:18.391 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 161 k 2020-09-08T16:44:18.391 INFO:teuthology.orchestra.run.smithi120.stdout:Installed size: 608 k 2020-09-08T16:44:18.391 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2020-09-08T16:44:18.559 INFO:teuthology.orchestra.run.smithi120.stdout:rbd-nbd-16.0.0-5272.gdce4d291bd4.el8.x86_64.rpm 951 kB/s | 161 kB 00:00 2020-09-08T16:44:18.560 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2020-09-08T16:44:18.560 INFO:teuthology.orchestra.run.smithi120.stdout:Total 945 kB/s | 161 kB 00:00 2020-09-08T16:44:18.562 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2020-09-08T16:44:18.574 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2020-09-08T16:44:18.574 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2020-09-08T16:44:18.622 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2020-09-08T16:44:18.623 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2020-09-08T16:44:18.776 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2020-09-08T16:44:19.000 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:19.207 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-nbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:19.452 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 1/1 2020-09-08T16:44:19.453 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:19.453 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2020-09-08T16:44:19.453 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:16.0.0-5272.gdce4d291bd4.el8.x86_64 2020-09-08T16:44:19.453 INFO:teuthology.orchestra.run.smithi120.stdout: 2020-09-08T16:44:19.454 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2020-09-08T16:44:19.729 DEBUG:teuthology.parallel:result is None 2020-09-08T16:44:19.731 INFO:teuthology.orchestra.run.smithi120:> sudo lsb_release -is 2020-09-08T16:44:19.768 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS 2020-09-08T16:44:19.769 DEBUG:teuthology.misc:System to be installed: CentOS 2020-09-08T16:44:19.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=dce4d291bd4243c9802c17d67b85a6da6bbc0f8f 2020-09-08T16:44:19.942 INFO:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-09-08T16:44:19.961 INFO:teuthology.orchestra.run.smithi120.stdout:16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:44:19.962 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-5272.gdce4d291bd4.el8 2020-09-08T16:44:19.962 INFO:teuthology.task.install:The correct ceph version 16.0.0-5272.gdce4d291bd4 is installed. 2020-09-08T16:44:19.963 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-09-08T16:44:19.964 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-09-08T16:44:19.964 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-09-08T16:44:20.013 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-09-08T16:44:20.026 INFO:teuthology.orchestra.run.smithi120.stderr:24+1 records in 2020-09-08T16:44:20.027 INFO:teuthology.orchestra.run.smithi120.stderr:24+1 records out 2020-09-08T16:44:20.027 INFO:teuthology.orchestra.run.smithi120.stderr:12667 bytes (13 kB, 12 KiB) copied, 6.6494e-05 s, 190 MB/s 2020-09-08T16:44:20.028 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-09-08T16:44:20.028 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-09-08T16:44:20.029 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2020-09-08T16:44:20.081 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-09-08T16:44:20.093 INFO:teuthology.orchestra.run.smithi120.stderr:5+1 records in 2020-09-08T16:44:20.094 INFO:teuthology.orchestra.run.smithi120.stderr:5+1 records out 2020-09-08T16:44:20.094 INFO:teuthology.orchestra.run.smithi120.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 3.4428e-05 s, 88.8 MB/s 2020-09-08T16:44:20.095 INFO:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-09-08T16:44:20.162 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-09-08T16:44:20.162 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-09-08T16:44:20.163 INFO:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2020-09-08T16:44:20.215 INFO:teuthology.orchestra.run.smithi120.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-09-08T16:44:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records in 2020-09-08T16:44:20.228 INFO:teuthology.orchestra.run.smithi120.stderr:0+1 records out 2020-09-08T16:44:20.229 INFO:teuthology.orchestra.run.smithi120.stderr:299 bytes copied, 3.009e-05 s, 9.9 MB/s 2020-09-08T16:44:20.230 INFO:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-09-08T16:44:20.297 INFO:teuthology.run_tasks:Running task workunit... 2020-09-08T16:44:20.319 INFO:tasks.workunit:Pulling workunits from ref dce4d291bd4243c9802c17d67b85a6da6bbc0f8f 2020-09-08T16:44:20.321 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-09-08T16:44:20.322 INFO:tasks.workunit:timeout=3h 2020-09-08T16:44:20.322 INFO:tasks.workunit:cleanup=True 2020-09-08T16:44:20.323 INFO:teuthology.orchestra.run.smithi120:> stat -- /home/ubuntu/cephtest/mnt.0 2020-09-08T16:44:20.351 INFO:teuthology.orchestra.run.smithi120.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-09-08T16:44:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-09-08T16:44:20.352 INFO:teuthology.orchestra.run.smithi120:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-09-08T16:44:20.405 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-09-08T16:44:20.405 INFO:teuthology.orchestra.run.smithi120:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-09-08T16:44:20.459 INFO:teuthology.orchestra.run.smithi120:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout dce4d291bd4243c9802c17d67b85a6da6bbc0f8f 2020-09-08T16:44:20.762 INFO:tasks.workunit.client.0.smithi120.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-09-08T16:45:08.610 INFO:tasks.workunit.client.0.smithi120.stderr:Note: checking out 'dce4d291bd4243c9802c17d67b85a6da6bbc0f8f'. 2020-09-08T16:45:24.718 INFO:tasks.workunit.client.0.smithi120.stderr: 2020-09-08T16:45:24.719 INFO:tasks.workunit.client.0.smithi120.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-09-08T16:45:24.719 INFO:tasks.workunit.client.0.smithi120.stderr:changes and commit them, and you can discard any commits you make in this 2020-09-08T16:45:24.719 INFO:tasks.workunit.client.0.smithi120.stderr:state without impacting any branches by performing another checkout. 2020-09-08T16:45:24.719 INFO:tasks.workunit.client.0.smithi120.stderr: 2020-09-08T16:45:24.720 INFO:tasks.workunit.client.0.smithi120.stderr:If you want to create a new branch to retain commits you create, you may 2020-09-08T16:45:24.720 INFO:tasks.workunit.client.0.smithi120.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-09-08T16:45:24.720 INFO:tasks.workunit.client.0.smithi120.stderr: 2020-09-08T16:45:24.721 INFO:tasks.workunit.client.0.smithi120.stderr: git checkout -b 2020-09-08T16:45:24.721 INFO:tasks.workunit.client.0.smithi120.stderr: 2020-09-08T16:45:24.721 INFO:tasks.workunit.client.0.smithi120.stderr:HEAD is now at dce4d291bd4 Merge branch 'wip-osd-pass-by-ref' of https://github.com/tchaikov/ceph into wip-kefu-testing-2020-09-08-0934 2020-09-08T16:45:24.723 INFO:teuthology.orchestra.run.smithi120:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-09-08T16:45:24.743 INFO:teuthology.orchestra.run.smithi120:> set -ex 2020-09-08T16:45:24.743 INFO:teuthology.orchestra.run.smithi120:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-09-08T16:45:24.798 INFO:teuthology.orchestra.run.smithi120.stderr:+ dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-09-08T16:45:24.800 INFO:teuthology.orchestra.run.smithi120.stdout:special/ceph_objectstore_tool.pyspecial/test-failure.shcrush/crush-classes.shcrush/crush-choose-args.shceph-helpers.shosd/osd-fast-mark-down.shosd/osd-recovery-prio.shosd/osd-backfill-prio.shosd/osd-dup.shosd/osd-copy-from.shosd/osd-reuse-id.shosd/osd-recovery-stats.shosd/osd-config.shosd/divergent-priors.shosd/pg-split-merge.shosd/osd-backfill-stats.shosd/repro_long_log.shosd/osd-reactivate.shosd/osd-backfill-space.shosd/ec-error-rollforward.shosd/osd-recovery-space.shosd/osd-backfill-recovery-log.shosd/repeer-on-acting-back.shosd/osd-bluefs-volume-ops.shosd/bad-inc-map.shosd/osd-force-create-pg.shosd/osd-bench.shosd/osd-rep-recov-eio.shosd/osd-markdown.shmisc/ok-to-stop.shmisc/network-ping.shmisc/test-ceph-helpers.shmisc/rados-striper.shmon/mon-scrub.shmon/mon-last-epoch-clean.shmon/osd-pool-df.shmon/test_pool_quota.shmon/osd-df.shmon/osd-pool-create.shmon/mkfs.shmon/health-mute.shmon/mon-created-time.shmon/osd-crush.shmon/osd-erasure-code-profile.shmon/misc.shmon/mon-bind.shmon/mon-osdmap-prune.shmon/mon-seesaw.shmon/mon-ping.shmon/mon-handle-forward.shscrub/osd-scrub-snaps.shscrub/osd-unexpected-clone.shscrub/osd-scrub-dump.shscrub/osd-recovery-scrub.shscrub/osd-scrub-repair.shscrub/osd-scrub-test.shmgr/balancer.sherasure-code/test-erasure-code-plugins.sherasure-code/test-erasure-eio.sherasure-code/test-erasure-code.sh 2020-09-08T16:45:24.800 INFO:teuthology.orchestra.run.smithi120.stderr:2+1 records in 2020-09-08T16:45:24.801 INFO:teuthology.orchestra.run.smithi120.stderr:2+1 records out 2020-09-08T16:45:24.801 INFO:teuthology.orchestra.run.smithi120.stderr:1404 bytes (1.4 kB, 1.4 KiB) copied, 3.918e-05 s, 35.8 MB/s 2020-09-08T16:45:24.801 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2020-09-08T16:45:24.802 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2020-09-08T16:45:24.803 INFO:teuthology.orchestra.run.smithi120:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dce4d291bd4243c9802c17d67b85a6da6bbc0f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2020-09-08T16:45:24.875 INFO:tasks.workunit.client.0.smithi120.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2020-09-08T16:45:24.876 INFO:tasks.workunit.client.0.smithi120.stderr:++ TIMEOUT=300 2020-09-08T16:45:24.876 INFO:tasks.workunit.client.0.smithi120.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2020-09-08T16:45:24.876 INFO:tasks.workunit.client.0.smithi120.stderr:++ MAX_TIMEOUT=15 2020-09-08T16:45:24.876 INFO:tasks.workunit.client.0.smithi120.stderr:++ PG_NUM=4 2020-09-08T16:45:24.877 INFO:tasks.workunit.client.0.smithi120.stderr:++ TMPDIR=/tmp 2020-09-08T16:45:24.878 INFO:tasks.workunit.client.0.smithi120.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2020-09-08T16:45:24.879 INFO:tasks.workunit.client.0.smithi120.stderr:++ TESTDIR=/home/ubuntu/cephtest 2020-09-08T16:45:24.879 INFO:tasks.workunit.client.0.smithi120.stderr:++ type xmlstarlet 2020-09-08T16:45:24.879 INFO:tasks.workunit.client.0.smithi120.stderr:++ XMLSTARLET=xmlstarlet 2020-09-08T16:45:24.880 INFO:tasks.workunit.client.0.smithi120.stderr:+++ uname 2020-09-08T16:45:24.881 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' Linux = FreeBSD ']' 2020-09-08T16:45:24.881 INFO:tasks.workunit.client.0.smithi120.stderr:++ SED=sed 2020-09-08T16:45:24.882 INFO:tasks.workunit.client.0.smithi120.stderr:++ AWK=awk 2020-09-08T16:45:24.882 INFO:tasks.workunit.client.0.smithi120.stderr:+++ stty -a 2020-09-08T16:45:24.882 INFO:tasks.workunit.client.0.smithi120.stderr:+++ head -1 2020-09-08T16:45:24.882 INFO:tasks.workunit.client.0.smithi120.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2020-09-08T16:45:24.883 INFO:tasks.workunit.client.0.smithi120.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-09-08T16:45:24.883 INFO:tasks.workunit.client.0.smithi120.stderr:++ termwidth= 2020-09-08T16:45:24.883 INFO:tasks.workunit.client.0.smithi120.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2020-09-08T16:45:24.883 INFO:tasks.workunit.client.0.smithi120.stderr:++ DIFFCOLOPTS='-y ' 2020-09-08T16:45:24.883 INFO:tasks.workunit.client.0.smithi120.stderr:++ KERNCORE=kernel.core_pattern 2020-09-08T16:45:24.884 INFO:tasks.workunit.client.0.smithi120.stderr:++ EXTRA_OPTS= 2020-09-08T16:45:24.887 INFO:tasks.workunit.client.0.smithi120.stderr:++ test '' = TESTS 2020-09-08T16:45:24.887 INFO:tasks.workunit.client.0.smithi120.stderr:++ uname -m 2020-09-08T16:45:24.887 INFO:tasks.workunit.client.0.smithi120.stderr:+ arch=x86_64 2020-09-08T16:45:24.887 INFO:tasks.workunit.client.0.smithi120.stderr:+ case $arch in 2020-09-08T16:45:24.888 INFO:tasks.workunit.client.0.smithi120.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2020-09-08T16:45:24.888 INFO:tasks.workunit.client.0.smithi120.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2020-09-08T16:45:24.888 INFO:tasks.workunit.client.0.smithi120.stderr:+ plugins=(jerasure shec lrc isa) 2020-09-08T16:45:24.888 INFO:tasks.workunit.client.0.smithi120.stderr:+ main test-erasure-code-plugins 2020-09-08T16:45:24.889 INFO:tasks.workunit.client.0.smithi120.stderr:+ local dir=td/test-erasure-code-plugins 2020-09-08T16:45:24.889 INFO:tasks.workunit.client.0.smithi120.stderr:+ shift 2020-09-08T16:45:24.889 INFO:tasks.workunit.client.0.smithi120.stderr:+ shopt -s -o xtrace 2020-09-08T16:45:24.889 INFO:tasks.workunit.client.0.smithi120.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-09-08T16:45:24.889 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-09-08T16:45:24.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-09-08T16:45:24.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-09-08T16:45:24.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-09-08T16:45:24.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-09-08T16:45:24.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-09-08T16:45:24.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-09-08T16:45:24.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-09-08T16:45:24.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/test-erasure-code-plugins 2020-09-08T16:45:24.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:24.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2020-09-08T16:45:24.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2020-09-08T16:45:24.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2020-09-08T16:45:24.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2020-09-08T16:45:24.893 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2020-09-08T16:45:24.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none ' 2020-09-08T16:45:24.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2020-09-08T16:45:24.896 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2020-09-08T16:45:24.896 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-09-08T16:45:24.897 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2020-09-08T16:45:24.897 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_preload_no_warning 2020-09-08T16:45:24.897 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_preload_no_warning_default 2020-09-08T16:45:24.897 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_preload_warning' 2020-09-08T16:45:24.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2020-09-08T16:45:24.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2020-09-08T16:45:24.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:24.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2020-09-08T16:45:24.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:24.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:45:24.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:24.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:45:24.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:45:24.900 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:45:24.900 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:45:24.900 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:45:24.900 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:45:24.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:45:24.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:45:24.901 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:45:24.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:45:24.902 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:45:24.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:45:24.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:45:24.905 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:45:24.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:45:24.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:45:24.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:45:24.906 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:45:24.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:45:24.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:45:24.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:45:24.907 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:45:24.907 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:24.907 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:24.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:45:24.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:45:24.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:45:24.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:45:24.908 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:45:24.908 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:24.909 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:24.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:45:24.911 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:45:24.911 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-09-08T16:45:24.911 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-09-08T16:45:24.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:45:24.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:45:24.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2020-09-08T16:45:24.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:24.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:45:24.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:45:24.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:45:24.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:45:24.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2020-09-08T16:45:25.221 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:45:25.222 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:45:25.222 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:45:25.222 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:45:25.223 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:25.223 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:25.223 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:25.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T16:45:25.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:45:25.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:45:25.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:45:25.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:45:25.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:45:25.419 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:45:25.419 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:45:25.419 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:45:25.419 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:45:25.420 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:45:25.420 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:25.420 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:25.420 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:45:25.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:45:25.421 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:45:25.515 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:45:25.515 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:45:25.515 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:45:25.516 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:45:25.516 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:45:25.516 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:45:25.516 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:45:25.517 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:45:25.517 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:45:25.517 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:25.517 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:25.518 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:45:25.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:45:25.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:45:25.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:45:25.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:25.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:45:25.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:45:25.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:45:25.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:45:25.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:45:26.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:45:26.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:45:26.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:45:26.510 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:45:26.510 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:26.511 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:26.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:26.512 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:45:26.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:45:26.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2020-09-08T16:45:26.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2020-09-08T16:45:26.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2020-09-08T16:45:26.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:26.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:45:26.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:45:26.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:45:26.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:45:26.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:45:26.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:45:26.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:45:26.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:45:26.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:45:26.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:45:26.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:45:26.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:45:26.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:45:26.552 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:45:26.553 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:45:26.553 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:45:26.553 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:45:26.554 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:26.554 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:26.554 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:26.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:26.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:45:26.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:45:26.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:45:26.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:45:26.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:45:26.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:45:26.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:45:26.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:45:26.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:45:26.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:45:26.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:45:26.558 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:45:26.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7f7abdf2-5ba1-4992-8100-4dadc0b3f452 2020-09-08T16:45:26.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 7f7abdf2-5ba1-4992-8100-4dadc0b3f452' 2020-09-08T16:45:26.559 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:45:26.560 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 7f7abdf2-5ba1-4992-8100-4dadc0b3f452 2020-09-08T16:45:26.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAmtVdf8BsrIRAAvkluDjLAFHxhkdsVKncGkQ== 2020-09-08T16:45:26.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAmtVdf8BsrIRAAvkluDjLAFHxhkdsVKncGkQ=="}' 2020-09-08T16:45:26.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7f7abdf2-5ba1-4992-8100-4dadc0b3f452 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:45:26.876 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:45:26.885 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:45:26.886 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAmtVdf8BsrIRAAvkluDjLAFHxhkdsVKncGkQ== --osd-uuid 7f7abdf2-5ba1-4992-8100-4dadc0b3f452 2020-09-08T16:45:26.907 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:26.911+0000 7f63cccbeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:26.909 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:26.913+0000 7f63cccbeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:26.910 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:26.915+0000 7f63cccbeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:26.911 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:26.915+0000 7f63cccbeec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:45:26.911 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:26.915+0000 7f63cccbeec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:45:26.911 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:26.915+0000 7f63cccbeec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:45:27.001 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:27.006+0000 7f63cccbeec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:45:31.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:45:31.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:45:31.310 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:45:31.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:45:31.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:45:31.798 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:45:31.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:45:31.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:45:31.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:45:31.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:45:31.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:45:31.819 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:31.820+0000 7f20becfdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:31.819 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:31.822+0000 7f20becfdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:31.820 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:31.823+0000 7f20becfdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:32.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:45:32.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:45:32.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:45:32.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:45:32.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:45:32.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:45:32.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:45:32.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:45:32.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:45:32.165 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:45:32.355 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:32.359+0000 7f20becfdec0 -1 Falling back to public interface 2020-09-08T16:45:32.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:45:33.205 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:33.209+0000 7f20becfdec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:45:33.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:45:33.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:45:33.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:45:33.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:45:33.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:45:33.532 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:45:33.949 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/18175,v1:127.0.0.1:6803/18175] [v2:127.0.0.1:6804/18175,v1:127.0.0.1:6805/18175] exists,up 7f7abdf2-5ba1-4992-8100-4dadc0b3f452 2020-09-08T16:45:33.950 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 5c516bcf-5807-4868-ab5d-6169b0321764 2020-09-08T16:45:33.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:45:33.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:45:33.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:45:33.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2020-09-08T16:45:33.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2020-09-08T16:45:33.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:33.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:45:33.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T16:45:33.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:45:33.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/1 2020-09-08T16:45:33.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:45:33.953 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:45:33.953 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:45:33.953 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:45:33.953 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2020-09-08T16:45:33.953 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2020-09-08T16:45:33.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:45:33.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:45:33.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:45:33.954 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:45:33.954 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:45:33.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:45:33.955 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:45:33.955 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:33.955 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:33.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:33.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:33.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:45:33.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:45:33.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:45:33.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:45:33.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:45:33.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:45:33.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:45:33.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:45:33.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:45:33.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:45:33.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/1 2020-09-08T16:45:33.958 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:45:33.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5c516bcf-5807-4868-ab5d-6169b0321764 2020-09-08T16:45:33.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 5c516bcf-5807-4868-ab5d-6169b0321764' 2020-09-08T16:45:33.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:45:33.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAttVdfJJCoORAAtCAJUodyMGIB+gNakhnSmA== 2020-09-08T16:45:33.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAttVdfJJCoORAAtCAJUodyMGIB+gNakhnSmA=="}' 2020-09-08T16:45:33.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5c516bcf-5807-4868-ab5d-6169b0321764 -i td/test-erasure-code-plugins/1/new.json 2020-09-08T16:45:34.393 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:45:34.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/1/new.json 2020-09-08T16:45:34.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAttVdfJJCoORAAtCAJUodyMGIB+gNakhnSmA== --osd-uuid 5c516bcf-5807-4868-ab5d-6169b0321764 2020-09-08T16:45:34.426 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:34.429+0000 7fc30733aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:34.426 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:34.431+0000 7fc30733aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:34.427 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:34.432+0000 7fc30733aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:34.428 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:34.433+0000 7fc30733aec0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2020-09-08T16:45:34.429 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:34.433+0000 7fc30733aec0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2020-09-08T16:45:34.429 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:34.433+0000 7fc30733aec0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2020-09-08T16:45:34.469 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:34.474+0000 7fc30733aec0 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2020-09-08T16:45:41.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2020-09-08T16:45:41.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:45:41.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T16:45:41.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:45:41.138 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T16:45:41.629 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T16:45:41.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T16:45:41.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:45:41.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:45:41.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:45:41.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:45:41.648 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:41.652+0000 7fef810a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:41.649 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:41.654+0000 7fef810a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:41.651 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:41.656+0000 7fef810a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:41.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T16:45:42.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:45:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T16:45:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:45:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:45:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:45:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:45:42.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:45:42.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:45:42.025 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:45:42.167 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:42.172+0000 7fef810a5ec0 -1 Falling back to public interface 2020-09-08T16:45:42.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:45:42.975 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:42.980+0000 7fef810a5ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T16:45:43.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:45:43.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:45:43.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:45:43.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:45:43.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:45:43.357 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:45:43.732 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/19177,v1:127.0.0.1:6811/19177] [v2:127.0.0.1:6812/19177,v1:127.0.0.1:6813/19177] exists,up 5c516bcf-5807-4868-ab5d-6169b0321764 2020-09-08T16:45:43.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:45:43.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:45:43.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:45:43.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2020-09-08T16:45:43.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2020-09-08T16:45:43.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:45:43.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:45:43.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T16:45:43.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:45:43.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/2 2020-09-08T16:45:43.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:45:43.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:45:43.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:45:43.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:45:43.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2020-09-08T16:45:43.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2020-09-08T16:45:43.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:45:43.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:45:43.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:45:43.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:45:43.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:45:43.737 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:45:43.737 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:45:43.737 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:45:43.737 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:45:43.737 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:43.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:45:43.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:45:43.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:45:43.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:45:43.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:45:43.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:45:43.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:45:43.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:45:43.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:45:43.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:45:43.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:45:43.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/2 2020-09-08T16:45:43.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:45:43.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=243dbec7-9202-4488-b006-0cc53a72cbb9 2020-09-08T16:45:43.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 243dbec7-9202-4488-b006-0cc53a72cbb9' 2020-09-08T16:45:43.751 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:45:43.751 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 243dbec7-9202-4488-b006-0cc53a72cbb9 2020-09-08T16:45:43.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA3tVdfX67KLBAACoihOC58uFNacR1qilHnAw== 2020-09-08T16:45:43.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA3tVdfX67KLBAACoihOC58uFNacR1qilHnAw=="}' 2020-09-08T16:45:43.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 243dbec7-9202-4488-b006-0cc53a72cbb9 -i td/test-erasure-code-plugins/2/new.json 2020-09-08T16:45:44.244 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:45:44.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/2/new.json 2020-09-08T16:45:44.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA3tVdfX67KLBAACoihOC58uFNacR1qilHnAw== --osd-uuid 243dbec7-9202-4488-b006-0cc53a72cbb9 2020-09-08T16:45:44.275 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:44.280+0000 7f263e628ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:44.278 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:44.281+0000 7f263e628ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:44.278 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:44.283+0000 7f263e628ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:44.279 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:44.283+0000 7f263e628ec0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2020-09-08T16:45:44.279 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:44.283+0000 7f263e628ec0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2020-09-08T16:45:44.279 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:44.283+0000 7f263e628ec0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2020-09-08T16:45:44.316 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:44.321+0000 7f263e628ec0 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2020-09-08T16:45:49.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2020-09-08T16:45:49.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:45:49.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T16:45:49.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:45:49.196 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T16:45:49.684 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T16:45:49.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T16:45:49.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:45:49.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:45:49.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:45:49.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:45:49.706 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:49.708+0000 7f232a2cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:49.707 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:49.710+0000 7f232a2cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:49.707 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:49.712+0000 7f232a2cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:45:50.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T16:45:50.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:45:50.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T16:45:50.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:45:50.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:45:50.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:45:50.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:45:50.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:45:50.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:45:50.052 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:45:50.224 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:50.230+0000 7f232a2cfec0 -1 Falling back to public interface 2020-09-08T16:45:50.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:45:51.044 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:45:51.048+0000 7f232a2cfec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T16:45:51.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:45:51.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:45:51.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:45:51.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:45:51.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:45:51.435 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:45:51.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:45:52.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:45:52.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:45:52.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:45:52.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:45:52.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:45:52.799 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:45:53.166 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/20171,v1:127.0.0.1:6819/20171] [v2:127.0.0.1:6820/20171,v1:127.0.0.1:6821/20171] exists,up 243dbec7-9202-4488-b006-0cc53a72cbb9 2020-09-08T16:45:53.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:45:53.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:45:53.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:45:53.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2020-09-08T16:45:53.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2020-09-08T16:45:53.522 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' does not exist 2020-09-08T16:45:53.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2020-09-08T16:45:53.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T16:45:54.037 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T16:45:54.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:45:55.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2020-09-08T16:45:56.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2020-09-08T16:45:56.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:45:56.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:45:56.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:45:56.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:45:56.570 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:45:56.570 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:45:56.570 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:45:56.570 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:45:56.570 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:45:56.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:45:56.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:45:56.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:45:56.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:45:56.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:45:56.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:45:57.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:45:57.037 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:45:57.038 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T16:45:57.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:45:57.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:45:57.038 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:45:57.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2020-09-08T16:45:57.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2020-09-08T16:45:57.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2020-09-08T16:45:57.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:45:57.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:45:57.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705669 2020-09-08T16:45:57.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705669 2020-09-08T16:45:57.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2020-09-08T16:45:57.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:45:57.290 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:45:57.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574852 2020-09-08T16:45:57.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574852 2020-09-08T16:45:57.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-55834574852' 2020-09-08T16:45:57.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:45:57.412 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-09-08T16:45:57.412 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:45:57.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:45:57.413 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2020-09-08T16:45:57.413 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:45:57.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2020-09-08T16:45:57.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-09-08T16:45:57.413 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 21474836487 2020-09-08T16:45:57.414 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:45:57.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2020-09-08T16:45:57.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:45:58.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T16:45:58.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:45:59.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2020-09-08T16:45:59.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:46:00.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 299 -eq 0 ']' 2020-09-08T16:46:00.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:46:00.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2020-09-08T16:46:00.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:46:00.521 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705669 2020-09-08T16:46:00.521 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:46:00.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:46:00.522 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705669 2020-09-08T16:46:00.522 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:46:00.523 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 38654705669 2020-09-08T16:46:00.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705669 2020-09-08T16:46:00.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-09-08T16:46:00.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:46:00.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705669 -lt 38654705669 2020-09-08T16:46:00.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:46:00.897 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-55834574852 2020-09-08T16:46:00.897 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:46:00.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:46:00.898 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-55834574852 2020-09-08T16:46:00.899 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:46:00.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574852 2020-09-08T16:46:00.900 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2020-09-08T16:46:00.900 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 55834574852 2020-09-08T16:46:00.900 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:46:01.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574853 -lt 55834574852 2020-09-08T16:46:01.381 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:46:01.382 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:46:01.382 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:46:01.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 4 == 0 2020-09-08T16:46:01.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:46:01.868 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:46:01.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:46:01.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:46:01.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:46:01.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:46:01.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:46:02.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:46:02.223 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:46:02.224 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:46:02.224 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:46:02.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 4 2020-09-08T16:46:02.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:46:02.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:46:02.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2020-09-08T16:46:02.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2020-09-08T16:46:03.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2020-09-08T16:46:03.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:03.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:03.288 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:03.288 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:03.288 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:03.289 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:03.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2020-09-08T16:46:03.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:03.366 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:03.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:03.376 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:02.976+0000 7fcb05487700 0 mon.a@0(leader).osd e22 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:03.376 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:03.281+0000 7fcb05487700 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:03.376 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:03.281+0000 7fcb05487700 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:03.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2020-09-08T16:46:03.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2020-09-08T16:46:04.075 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2020-09-08T16:46:04.075 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:04.076 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:04.076 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:04.076 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:04.076 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:04.077 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:04.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2020-09-08T16:46:04.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:04.154 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:04.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:04.164 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:03.643+0000 7fcb05487700 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:04.164 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:04.068+0000 7fcb05487700 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:04.164 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:04.068+0000 7fcb05487700 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:04.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2020-09-08T16:46:04.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2020-09-08T16:46:04.679 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2020-09-08T16:46:04.679 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:04.679 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:04.680 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:04.680 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:04.680 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:04.680 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:04.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2020-09-08T16:46:04.681 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:04.757 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:04.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:04.767 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:04.432+0000 7fcb05487700 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:04.767 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:04.671+0000 7fcb05487700 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:04.767 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:04.671+0000 7fcb05487700 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2020-09-08T16:46:04.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2020-09-08T16:46:04.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2020-09-08T16:46:05.300 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2020-09-08T16:46:05.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:05.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:05.301 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:05.301 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:05.302 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:05.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:05.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2020-09-08T16:46:05.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:05.381 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:05.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:05.389 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:05.030+0000 7fcb05487700 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2020-09-08T16:46:05.389 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:05.293+0000 7fcb05487700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2020-09-08T16:46:05.389 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:05.293+0000 7fcb05487700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2020-09-08T16:46:05.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2020-09-08T16:46:05.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2020-09-08T16:46:06.039 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2020-09-08T16:46:06.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:06.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:06.040 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:06.041 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:06.041 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:06.041 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:06.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2020-09-08T16:46:06.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:06.119 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:06.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:06.129 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:05.653+0000 7fcb05487700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2020-09-08T16:46:06.129 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:06.032+0000 7fcb05487700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2020-09-08T16:46:06.130 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:06.032+0000 7fcb05487700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2020-09-08T16:46:06.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2020-09-08T16:46:06.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2020-09-08T16:46:06.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2020-09-08T16:46:06.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:06.702 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:06.702 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:06.702 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:06.702 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:06.702 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:06.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2020-09-08T16:46:06.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:06.781 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:06.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:06.791 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:06.396+0000 7fcb05487700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2020-09-08T16:46:06.791 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:06.694+0000 7fcb05487700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2020-09-08T16:46:06.792 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:06.694+0000 7fcb05487700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2020-09-08T16:46:06.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:46:06.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:06.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:06.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:06.797 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:06.798 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:06.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:06.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:06.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:06.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:06.900 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:06.900 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:06.901 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:06.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:06.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:06.902 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:06.903 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:06.903 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:06.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:06.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:06.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:06.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:06.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:06.909 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:06.910 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:06.910 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:06.910 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:06.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:06.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:06.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2020-09-08T16:46:06.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2020-09-08T16:46:06.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:06.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2020-09-08T16:46:06.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2020-09-08T16:46:06.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:06.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:46:06.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:06.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:06.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:06.915 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:06.915 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:06.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:06.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:06.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:06.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:06.917 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:06.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:06.918 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:06.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:06.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:06.918 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:06.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:06.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:06.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:06.919 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:06.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:06.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:06.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:06.921 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:06.921 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:06.921 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:06.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:06.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:06.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:06.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:46:06.924 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:46:06.924 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:06.924 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:06.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:46:06.924 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:46:06.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:46:06.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:46:06.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:46:06.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2020-09-08T16:46:06.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:06.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:46:06.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:46:06.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:46:06.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:46:06.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2020-09-08T16:46:07.250 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:46:07.251 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:07.251 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:07.251 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:07.251 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:07.251 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:07.252 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:07.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=jerasure 2020-09-08T16:46:07.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:46:07.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:46:07.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:07.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:07.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:46:07.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:46:07.421 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:07.421 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:07.421 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:07.421 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:07.421 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:07.422 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:07.422 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:07.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:07.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:46:07.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:46:07.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:07.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:07.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:46:07.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:46:07.509 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:07.509 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:07.509 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:07.509 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:07.510 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:07.510 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:07.510 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:07.510 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:07.510 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:46:07.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:46:07.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:07.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:46:07.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:46:07.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:46:07.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:46:07.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:46:08.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:46:08.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:08.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:08.478 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:08.478 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:08.478 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:08.478 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:08.479 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:08.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:08.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2020-09-08T16:46:08.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:08.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:08.500 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:08.500 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:08.500 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:08.500 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:08.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:08.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:08.585 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:08.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2020-09-08T16:46:08.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:08.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:46:08.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:46:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:46:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:46:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:46:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:46:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:46:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:46:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:46:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:46:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:46:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:46:08.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:46:08.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:46:08.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:08.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:08.594 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:08.595 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:46:08.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:46:08.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:46:08.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:46:08.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:46:08.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:46:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:46:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:46:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:46:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2020-09-08T16:46:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:46:08.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:46:08.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=56b29c4d-5a98-4a68-b5e4-00321910d375 2020-09-08T16:46:08.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 56b29c4d-5a98-4a68-b5e4-00321910d375' 2020-09-08T16:46:08.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:46:08.599 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 56b29c4d-5a98-4a68-b5e4-00321910d375 2020-09-08T16:46:08.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBQtVdfhuDHJBAA1FCxTCS0D7j8nhAdNA9WaA== 2020-09-08T16:46:08.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBQtVdfhuDHJBAA1FCxTCS0D7j8nhAdNA9WaA=="}' 2020-09-08T16:46:08.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 56b29c4d-5a98-4a68-b5e4-00321910d375 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:08.961 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:08.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:08.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure --mkfs --key AQBQtVdfhuDHJBAA1FCxTCS0D7j8nhAdNA9WaA== --osd-uuid 56b29c4d-5a98-4a68-b5e4-00321910d375 2020-09-08T16:46:08.993 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:08.998+0000 7f5113248ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:08.995 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:09.000+0000 7f5113248ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:08.996 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:09.001+0000 7f5113248ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:08.996 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:09.001+0000 7f5113248ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:08.997 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:09.001+0000 7f5113248ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:08.997 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:09.001+0000 7f5113248ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:09.035 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:09.040+0000 7f5113248ec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:46:13.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:46:13.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:46:13.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:46:13.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:46:13.338 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:46:13.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:46:13.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure 2020-09-08T16:46:13.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:46:13.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:46:13.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:46:13.835 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:46:13.853 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:13.858+0000 7fc049093ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:13.855 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:13.860+0000 7fc049093ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:13.856 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:13.861+0000 7fc049093ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:14.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:46:14.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:46:14.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:46:14.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:46:14.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:46:14.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:14.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:46:14.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:14.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:14.203 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:14.372 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:14.377+0000 7fc049093ec0 -1 Falling back to public interface 2020-09-08T16:46:14.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:46:15.203 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:15.208+0000 7fc049093ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:46:15.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:46:15.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:15.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:46:15.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:15.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:15.583 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:46:15.954 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/22128,v1:127.0.0.1:6803/22128] [v2:127.0.0.1:6804/22128,v1:127.0.0.1:6805/22128] exists,up 56b29c4d-5a98-4a68-b5e4-00321910d375 2020-09-08T16:46:15.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:46:15.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:46:15.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:46:15.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2020-09-08T16:46:15.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:46:15.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:46:15.956 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:15.956 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:15.956 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:15.956 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:46:15.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:15.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:46:16.044 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:16.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:16.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:46:16.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:46:16.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:16.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:16.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:16.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:16.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:16.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:16.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:16.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:16.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:16.160 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:16.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:16.161 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:16.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:16.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:16.162 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:16.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:16.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:16.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:16.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:16.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:16.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:16.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:16.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:16.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:16.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:16.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:16.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:16.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:16.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2020-09-08T16:46:16.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2020-09-08T16:46:16.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:16.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:46:16.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:16.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:16.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:16.171 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:16.171 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:16.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:16.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:16.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:16.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:16.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:16.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:16.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:16.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:16.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:16.174 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:16.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:16.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:16.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:16.175 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:16.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:16.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:16.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:16.177 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:16.177 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:16.177 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:16.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:16.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:16.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:16.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:46:16.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:46:16.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:16.180 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:16.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:46:16.181 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:46:16.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:46:16.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:46:16.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:46:16.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2020-09-08T16:46:16.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:16.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:46:16.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:46:16.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:46:16.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:46:16.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2020-09-08T16:46:16.473 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:46:16.473 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:16.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:16.474 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:16.474 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:16.474 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:16.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:16.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=shec 2020-09-08T16:46:16.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:46:16.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:46:16.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:16.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:16.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:46:16.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:46:16.651 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:16.651 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:16.652 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:16.652 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:16.652 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:16.652 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:16.652 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:16.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:16.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:46:16.737 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:46:16.737 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:16.738 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:16.738 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:46:16.738 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:46:16.738 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:16.738 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:16.739 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:16.739 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:16.739 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:16.739 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:16.740 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:16.740 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:16.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:46:16.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:46:16.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:16.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:46:16.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:46:16.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:46:16.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:46:16.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:46:17.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:46:17.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:17.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:17.700 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:17.701 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:17.701 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:17.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:17.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:17.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:17.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2020-09-08T16:46:17.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:17.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:17.722 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:17.722 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:17.723 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:17.723 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:17.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:17.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:17.803 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:17.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2020-09-08T16:46:17.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:17.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:46:17.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:46:17.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:46:17.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:46:17.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:46:17.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:46:17.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:46:17.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:46:17.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:46:17.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:46:17.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:46:17.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:46:17.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:46:17.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:46:17.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:17.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:17.817 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:17.818 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:17.818 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:17.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:17.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:17.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:46:17.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:46:17.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:46:17.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:46:17.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:46:17.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:46:17.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:46:17.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:46:17.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:46:17.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2020-09-08T16:46:17.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:46:17.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:46:17.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d7a6f84c-b54d-4cf6-8121-7192432356e2 2020-09-08T16:46:17.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d7a6f84c-b54d-4cf6-8121-7192432356e2' 2020-09-08T16:46:17.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:46:17.822 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 d7a6f84c-b54d-4cf6-8121-7192432356e2 2020-09-08T16:46:17.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBZtVdfhd2YMRAA0x8MpEf2naKcPS/wIo/w2Q== 2020-09-08T16:46:17.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBZtVdfhd2YMRAA0x8MpEf2naKcPS/wIo/w2Q=="}' 2020-09-08T16:46:17.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d7a6f84c-b54d-4cf6-8121-7192432356e2 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:18.166 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:18.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:18.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec --mkfs --key AQBZtVdfhd2YMRAA0x8MpEf2naKcPS/wIo/w2Q== --osd-uuid d7a6f84c-b54d-4cf6-8121-7192432356e2 2020-09-08T16:46:18.197 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:18.202+0000 7f2580a28ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:18.199 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:18.204+0000 7f2580a28ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:18.200 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:18.205+0000 7f2580a28ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:18.201 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:18.206+0000 7f2580a28ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:18.201 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:18.206+0000 7f2580a28ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:18.202 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:18.206+0000 7f2580a28ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:18.316 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:18.321+0000 7f2580a28ec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:46:22.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:46:22.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:46:22.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:46:22.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:46:22.679 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:46:23.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:46:23.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec 2020-09-08T16:46:23.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:46:23.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:46:23.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:46:23.166 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:46:23.184 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:23.189+0000 7f751ceddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:23.186 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:23.190+0000 7f751ceddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:23.187 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:23.192+0000 7f751ceddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:23.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:46:23.533 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:46:23.533 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:46:23.533 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:46:23.533 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:46:23.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:23.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:46:23.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:23.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:23.534 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:23.703 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:23.708+0000 7f751ceddec0 -1 Falling back to public interface 2020-09-08T16:46:23.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:46:24.541 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:24.546+0000 7f751ceddec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:46:24.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:46:24.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:24.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:46:24.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:24.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:24.924 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:46:25.304 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/23396,v1:127.0.0.1:6803/23396] [v2:127.0.0.1:6804/23396,v1:127.0.0.1:6805/23396] exists,up d7a6f84c-b54d-4cf6-8121-7192432356e2 2020-09-08T16:46:25.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:46:25.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:46:25.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:46:25.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2020-09-08T16:46:25.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:46:25.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:46:25.305 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:25.306 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:25.306 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:25.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:46:25.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:25.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:46:25.392 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:25.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:25.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:46:25.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:46:25.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:25.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:25.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:25.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:25.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:25.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:25.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:25.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:25.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:25.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:25.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:25.510 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:25.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:25.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:25.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:25.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:25.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:25.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:25.515 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:25.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:25.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:25.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:25.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:25.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:25.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:25.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:25.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:25.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:25.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2020-09-08T16:46:25.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2020-09-08T16:46:25.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:25.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:46:25.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:25.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:25.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:25.521 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:25.521 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:25.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:25.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:25.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:25.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:25.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:25.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:25.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:25.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:25.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:25.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:25.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:25.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:25.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:25.525 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:25.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:25.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:25.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:25.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:25.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:25.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:25.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:25.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:25.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:25.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:46:25.542 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:46:25.542 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:25.543 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:25.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:46:25.543 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:46:25.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:46:25.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:46:25.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:46:25.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2020-09-08T16:46:25.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:25.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:46:25.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:46:25.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:46:25.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:46:25.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2020-09-08T16:46:25.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:46:25.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:25.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:25.829 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:25.829 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:25.829 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:25.829 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:25.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=lrc 2020-09-08T16:46:26.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:46:26.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:46:26.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:26.041 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:26.041 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:46:26.041 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:46:26.041 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:26.042 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:26.042 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:26.042 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:26.043 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:26.043 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:26.043 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:26.043 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:26.043 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:46:26.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:46:26.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:26.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:26.130 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:46:26.130 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:46:26.130 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:26.130 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:26.130 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:26.131 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:26.131 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:26.131 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:26.131 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:26.132 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:26.132 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:46:26.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:46:26.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:26.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:46:26.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:46:26.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:46:26.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:46:26.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:46:26.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:46:26.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:26.713 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:26.713 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:26.713 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:26.713 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:26.713 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:26.714 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:26.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:26.734 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2020-09-08T16:46:26.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:26.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:26.736 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:26.736 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:26.736 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:26.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:26.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:26.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:26.817 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:26.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2020-09-08T16:46:26.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:26.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:46:26.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:46:26.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:46:26.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:46:26.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:46:26.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:46:26.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:46:26.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:46:26.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:46:26.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:46:26.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:46:26.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:46:26.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:46:26.831 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:46:26.831 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:26.831 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:26.831 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:26.831 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:26.831 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:26.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:26.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:26.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:46:26.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:46:26.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:46:26.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:46:26.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:46:26.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:46:26.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:46:26.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:46:26.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:46:26.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2020-09-08T16:46:26.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:46:26.834 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:46:26.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=86f3382f-6564-467d-b4e2-c9589549bf62 2020-09-08T16:46:26.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 86f3382f-6564-467d-b4e2-c9589549bf62' 2020-09-08T16:46:26.835 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:46:26.836 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 86f3382f-6564-467d-b4e2-c9589549bf62 2020-09-08T16:46:26.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBitVdfbDh1MhAAneFXsJQvyojBfggzxR0rAg== 2020-09-08T16:46:26.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBitVdfbDh1MhAAneFXsJQvyojBfggzxR0rAg=="}' 2020-09-08T16:46:26.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 86f3382f-6564-467d-b4e2-c9589549bf62 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:27.163 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:27.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:27.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=lrc --mkfs --key AQBitVdfbDh1MhAAneFXsJQvyojBfggzxR0rAg== --osd-uuid 86f3382f-6564-467d-b4e2-c9589549bf62 2020-09-08T16:46:27.195 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:27.200+0000 7feb7ffc2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:27.196 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:27.201+0000 7feb7ffc2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:27.198 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:27.202+0000 7feb7ffc2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:27.198 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:27.203+0000 7feb7ffc2ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:27.199 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:27.203+0000 7feb7ffc2ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:27.199 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:27.203+0000 7feb7ffc2ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:27.286 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:27.285+0000 7feb7ffc2ec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:46:31.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:46:31.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:46:31.655 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:46:31.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:46:31.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:46:32.152 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:46:32.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:46:32.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=lrc 2020-09-08T16:46:32.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:46:32.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:46:32.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:46:32.172 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:32.177+0000 7fa237a26ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:32.223 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:32.227+0000 7fa237a26ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:32.223 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:32.228+0000 7fa237a26ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:32.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:46:32.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:46:32.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:46:32.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:46:32.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:46:32.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:32.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:46:32.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:32.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:32.536 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:32.739 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:32.744+0000 7fa237a26ec0 -1 Falling back to public interface 2020-09-08T16:46:32.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:46:33.346 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:33.351+0000 7fa237a26ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:46:33.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:46:33.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:33.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:46:33.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:33.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:33.896 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:46:34.286 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/24667,v1:127.0.0.1:6803/24667] [v2:127.0.0.1:6804/24667,v1:127.0.0.1:6805/24667] exists,up 86f3382f-6564-467d-b4e2-c9589549bf62 2020-09-08T16:46:34.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:46:34.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:46:34.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:46:34.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2020-09-08T16:46:34.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:46:34.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:46:34.288 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:34.288 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:34.288 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:34.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:46:34.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:34.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:46:34.375 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:34.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:34.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:46:34.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:46:34.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:34.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:34.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:34.387 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:34.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:34.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:34.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:34.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:34.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:34.491 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:34.491 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:34.492 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:34.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:34.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:34.493 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:34.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:34.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:34.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:34.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:34.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:34.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:34.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:34.498 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:34.498 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:34.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:34.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:34.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:34.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:34.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2020-09-08T16:46:34.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2020-09-08T16:46:34.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:34.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:46:34.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:34.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:34.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:34.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:34.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:34.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:34.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:34.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:34.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:34.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:34.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:34.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:34.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:34.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:34.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:34.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:34.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:34.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:34.507 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:34.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:34.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:34.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:34.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:34.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:34.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:34.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:34.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:34.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:34.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:46:34.510 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:46:34.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:34.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:34.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:46:34.512 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:46:34.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:46:34.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:46:34.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:46:34.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2020-09-08T16:46:34.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:34.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:46:34.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:46:34.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:46:34.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:46:34.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2020-09-08T16:46:34.768 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:46:34.768 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:34.768 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:34.769 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:34.769 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:34.769 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:34.769 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:34.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=isa 2020-09-08T16:46:34.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:46:34.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:46:34.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:34.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:34.969 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:46:34.969 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:46:34.969 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:34.969 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:34.970 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:34.970 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:34.970 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:34.970 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:34.970 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:34.971 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:34.971 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:46:35.056 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:46:35.056 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:35.057 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:35.057 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:46:35.057 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:46:35.057 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:35.058 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:35.058 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:35.058 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:35.058 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:35.058 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:35.059 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:35.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:35.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:46:35.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:46:35.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:35.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:46:35.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:46:35.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:46:35.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:46:35.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:46:36.026 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:46:36.027 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:36.027 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:36.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:36.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:36.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:36.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2020-09-08T16:46:36.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:36.048 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:36.048 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:36.049 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:36.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:36.131 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2020-09-08T16:46:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:36.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:46:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:46:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:46:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:46:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:46:36.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:46:36.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:46:36.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:46:36.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:46:36.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:46:36.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:46:36.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:46:36.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:46:36.142 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:46:36.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:36.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:36.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:36.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:36.144 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:36.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:36.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:36.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:46:36.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:46:36.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:46:36.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:46:36.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:46:36.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:46:36.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:46:36.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:46:36.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:46:36.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2020-09-08T16:46:36.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:46:36.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:46:36.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=96767954-03d5-4d50-8662-23ea9e66eb30 2020-09-08T16:46:36.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 96767954-03d5-4d50-8662-23ea9e66eb30' 2020-09-08T16:46:36.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:46:36.150 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 96767954-03d5-4d50-8662-23ea9e66eb30 2020-09-08T16:46:36.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBstVdf985lCRAAzERwK+90edjZr+Xtlwf1lA== 2020-09-08T16:46:36.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBstVdf985lCRAAzERwK+90edjZr+Xtlwf1lA=="}' 2020-09-08T16:46:36.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 96767954-03d5-4d50-8662-23ea9e66eb30 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:36.467 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:36.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:36.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=isa --mkfs --key AQBstVdf985lCRAAzERwK+90edjZr+Xtlwf1lA== --osd-uuid 96767954-03d5-4d50-8662-23ea9e66eb30 2020-09-08T16:46:36.498 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:36.503+0000 7f502899dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:36.500 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:36.506+0000 7f502899dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:36.501 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:36.507+0000 7f502899dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:36.502 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:36.507+0000 7f502899dec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:36.502 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:36.507+0000 7f502899dec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:36.502 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:36.507+0000 7f502899dec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:36.542 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:36.547+0000 7f502899dec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:46:41.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:46:41.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:46:41.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:46:41.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:46:41.020 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:46:41.518 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:46:41.518 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=isa 2020-09-08T16:46:41.518 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:46:41.518 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:46:41.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:46:41.519 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:46:41.538 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:41.543+0000 7fab11223ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:41.554 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:41.547+0000 7fab11223ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:41.554 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:41.548+0000 7fab11223ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:41.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:46:41.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:46:41.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:46:41.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:46:41.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:46:41.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:41.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:46:41.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:41.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:41.883 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:42.064 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:42.069+0000 7fab11223ec0 -1 Falling back to public interface 2020-09-08T16:46:42.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:46:42.869 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:42.874+0000 7fab11223ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:46:43.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:46:43.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:43.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:46:43.246 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:46:43.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:43.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:43.619 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25942,v1:127.0.0.1:6803/25942] [v2:127.0.0.1:6804/25942,v1:127.0.0.1:6805/25942] exists,up 96767954-03d5-4d50-8662-23ea9e66eb30 2020-09-08T16:46:43.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:46:43.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:46:43.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:46:43.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2020-09-08T16:46:43.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:46:43.621 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:46:43.621 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:43.621 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:43.621 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:43.621 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:46:43.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2020-09-08T16:46:43.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:46:43.708 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:43.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:43.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:46:43.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:46:43.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:43.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:43.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:43.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:43.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:43.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:43.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:43.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:43.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:43.824 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:43.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:43.825 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:43.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:43.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:43.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:43.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:43.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:43.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:43.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:43.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:43.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:43.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:43.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:43.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:43.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:43.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:43.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:43.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:43.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2020-09-08T16:46:43.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2020-09-08T16:46:43.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2020-09-08T16:46:43.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:43.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2020-09-08T16:46:43.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:43.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:46:43.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:43.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:43.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:43.837 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:43.837 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:43.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:43.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:43.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:43.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:43.840 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:43.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:43.840 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:43.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:43.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:43.841 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:43.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:43.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:43.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:43.849 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:43.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:43.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:43.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:43.853 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:43.853 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:43.853 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:43.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:43.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:43.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:43.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:46:43.854 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:46:43.855 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:43.855 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:43.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:46:43.855 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:46:43.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:46:43.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:46:43.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:46:43.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2020-09-08T16:46:43.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:43.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:46:43.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:46:43.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:46:43.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:46:43.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2020-09-08T16:46:44.128 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:46:44.128 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:44.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:44.129 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:44.129 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:44.129 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:44.130 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:44.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T16:46:44.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:46:44.298 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:46:44.298 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:44.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:44.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:46:44.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:46:44.299 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:44.299 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:44.300 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:44.300 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:44.300 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:44.300 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:44.300 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:44.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:44.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:46:44.384 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:46:44.385 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:44.385 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:44.385 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:46:44.385 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:46:44.386 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:44.386 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:44.386 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:44.386 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:44.387 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:44.387 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:44.387 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:44.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:44.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:46:44.472 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2020-09-08T16:46:44.472 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:44.472 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:44.472 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:44.473 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:44.473 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:44.473 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:44.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2020-09-08T16:46:44.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:44.551 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:44.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:46:44.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:44.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:46:44.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:46:44.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:46:44.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:46:44.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:46:45.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:46:45.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:45.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:45.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:45.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:45.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:45.073 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:45.073 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:45.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:45.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2020-09-08T16:46:45.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:45.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:46:45.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:46:45.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:46:45.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:46:45.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:46:45.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:46:45.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:46:45.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:46:45.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:46:45.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:46:45.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:46:45.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:46:45.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:46:45.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:46:45.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:45.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:45.099 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:45.100 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:45.100 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:45.100 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:45.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:45.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:46:45.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:46:45.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:46:45.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:46:45.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:46:45.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:46:45.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:46:45.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:46:45.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:46:45.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:46:45.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:46:45.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:46:45.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4a00cc56-c556-4df8-9d04-0b8ac2e97c58 2020-09-08T16:46:45.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 4a00cc56-c556-4df8-9d04-0b8ac2e97c58' 2020-09-08T16:46:45.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:46:45.105 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 4a00cc56-c556-4df8-9d04-0b8ac2e97c58 2020-09-08T16:46:45.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB1tVdfKfrTBhAA2mWeTjzkEbry68VYOMbfwQ== 2020-09-08T16:46:45.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB1tVdfKfrTBhAA2mWeTjzkEbry68VYOMbfwQ=="}' 2020-09-08T16:46:45.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4a00cc56-c556-4df8-9d04-0b8ac2e97c58 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:45.475 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:45.482 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:45.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB1tVdfKfrTBhAA2mWeTjzkEbry68VYOMbfwQ== --osd-uuid 4a00cc56-c556-4df8-9d04-0b8ac2e97c58 2020-09-08T16:46:45.504 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:45.509+0000 7f32263adec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:45.505 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:45.511+0000 7f32263adec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:45.508 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:45.513+0000 7f32263adec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:45.508 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:45.513+0000 7f32263adec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:45.508 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:45.513+0000 7f32263adec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:45.509 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:45.513+0000 7f32263adec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:45.540 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:45.545+0000 7f32263adec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:46:49.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:46:49.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:46:49.914 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:46:49.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:46:49.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:46:50.402 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:46:50.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:46:50.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:46:50.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:46:50.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:46:50.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:46:50.421 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:50.425+0000 7f0f096e2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:50.422 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:50.427+0000 7f0f096e2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:50.423 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:50.428+0000 7f0f096e2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:50.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:46:50.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:46:50.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:46:50.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:46:50.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:46:50.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:50.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:46:50.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:50.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:50.775 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:50.938 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:50.943+0000 7f0f096e2ec0 -1 Falling back to public interface 2020-09-08T16:46:51.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:46:51.762 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:51.767+0000 7f0f096e2ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:46:52.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:46:52.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:46:52.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:46:52.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:46:52.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:46:52.155 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:46:52.522 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/27210,v1:127.0.0.1:6803/27210] [v2:127.0.0.1:6804/27210,v1:127.0.0.1:6805/27210] exists,up 4a00cc56-c556-4df8-9d04-0b8ac2e97c58 2020-09-08T16:46:52.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:46:52.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:46:52.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:46:52.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2020-09-08T16:46:52.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:46:52.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:46:52.524 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:52.525 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:52.525 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:52.525 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:46:52.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2020-09-08T16:46:52.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:46:52.613 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:52.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:46:52.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:46:52.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2020-09-08T16:46:52.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:52.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:52.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:52.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:52.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:52.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:52.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:52.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:52.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:52.729 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:52.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:52.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:52.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:52.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:52.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:52.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:52.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:52.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:52.733 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:52.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:52.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:52.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:52.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:52.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:52.737 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:52.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:52.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:52.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:52.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2020-09-08T16:46:52.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2020-09-08T16:46:52.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2020-09-08T16:46:52.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:52.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2020-09-08T16:46:52.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2020-09-08T16:46:52.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:52.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:46:52.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:52.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:46:52.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:46:52.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:46:52.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:46:52.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:46:52.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:46:52.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:46:52.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:46:52.744 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:46:52.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:46:52.745 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:46:52.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:46:52.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:46:52.745 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:46:52.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:52.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:46:52.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:46:52.746 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:46:52.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:46:52.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:46:52.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:46:52.748 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:46:52.748 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:52.749 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:52.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:46:52.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:46:52.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:46:52.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:46:52.750 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:46:52.750 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:52.750 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:52.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:46:52.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:46:52.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:46:52.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:46:52.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:46:52.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2020-09-08T16:46:52.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:52.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:46:52.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:46:52.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:46:52.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:46:52.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2020-09-08T16:46:53.039 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:46:53.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:53.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:53.040 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:53.041 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:53.041 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:53.041 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:53.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=jerasure_generic 2020-09-08T16:46:53.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:46:53.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:46:53.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:53.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:53.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:46:53.220 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:46:53.220 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:53.220 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:53.221 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:53.221 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:53.221 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:53.222 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:53.222 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:53.222 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:53.223 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:46:53.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:46:53.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:46:53.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:46:53.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:46:53.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:46:53.306 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:46:53.306 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:53.306 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:53.306 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:53.307 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:53.307 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:53.307 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:53.308 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:46:53.308 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:46:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:46:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:46:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:46:53.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:46:53.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:46:53.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:46:54.257 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:46:54.257 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:54.258 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:54.258 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:54.258 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:54.258 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:54.258 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:54.259 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:54.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:46:54.278 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2020-09-08T16:46:54.279 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:46:54.279 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:46:54.279 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:46:54.279 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:54.280 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:54.280 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:46:54.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:46:54.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:46:54.364 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:46:54.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2020-09-08T16:46:54.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:46:54.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:46:54.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:46:54.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:46:54.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:46:54.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:46:54.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:46:54.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:46:54.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:46:54.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:46:54.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:46:54.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:46:54.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:46:54.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:46:54.375 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:46:54.375 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:46:54.376 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:46:54.376 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:46:54.377 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:46:54.378 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:46:54.378 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:54.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:46:54.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:46:54.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:46:54.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:46:54.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:46:54.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:46:54.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:46:54.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:46:54.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:46:54.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:46:54.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2020-09-08T16:46:54.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:46:54.381 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:46:54.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8eb5b3dc-cdf8-43ac-9d3f-88f7a99b7cf4 2020-09-08T16:46:54.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 8eb5b3dc-cdf8-43ac-9d3f-88f7a99b7cf4' 2020-09-08T16:46:54.384 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:46:54.384 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 8eb5b3dc-cdf8-43ac-9d3f-88f7a99b7cf4 2020-09-08T16:46:54.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB+tVdfTrI3FxAA4do7hIqZltG9NYJVsiDm9g== 2020-09-08T16:46:54.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB+tVdfTrI3FxAA4do7hIqZltG9NYJVsiDm9g=="}' 2020-09-08T16:46:54.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8eb5b3dc-cdf8-43ac-9d3f-88f7a99b7cf4 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:54.715 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:46:54.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:46:54.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQB+tVdfTrI3FxAA4do7hIqZltG9NYJVsiDm9g== --osd-uuid 8eb5b3dc-cdf8-43ac-9d3f-88f7a99b7cf4 2020-09-08T16:46:54.746 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:54.751+0000 7fedd1d3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:54.747 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:54.752+0000 7fedd1d3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:54.749 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:54.754+0000 7fedd1d3eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:54.750 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:54.755+0000 7fedd1d3eec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:54.750 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:54.755+0000 7fedd1d3eec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:54.750 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:54.755+0000 7fedd1d3eec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:46:54.814 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:54.818+0000 7fedd1d3eec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:46:59.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:46:59.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:46:59.314 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:46:59.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:46:59.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:46:59.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:46:59.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:46:59.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:46:59.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:46:59.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure_generic 2020-09-08T16:46:59.808 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:46:59.826 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:59.831+0000 7f18e7fb0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:59.828 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:59.833+0000 7f18e7fb0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:46:59.829 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:46:59.834+0000 7f18e7fb0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:00.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:47:00.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:47:00.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:47:00.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:47:00.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:47:00.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:00.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:47:00.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:00.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:00.176 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:00.347 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:00.352+0000 7f18e7fb0ec0 -1 Falling back to public interface 2020-09-08T16:47:00.533 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:47:01.172 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:01.177+0000 7f18e7fb0ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:47:01.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:47:01.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:01.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:47:01.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:01.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:01.536 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:01.928 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/28478,v1:127.0.0.1:6803/28478] [v2:127.0.0.1:6804/28478,v1:127.0.0.1:6805/28478] exists,up 8eb5b3dc-cdf8-43ac-9d3f-88f7a99b7cf4 2020-09-08T16:47:01.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:47:01.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:47:01.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:47:01.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2020-09-08T16:47:01.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:47:01.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:47:01.930 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:01.930 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:01.931 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:01.931 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:47:01.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:01.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:47:02.018 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:02.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:47:02.026 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:46:53.068+0000 7fd996bc46c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2020-09-08T16:47:02.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:47:02.028 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:00.353+0000 7f18e7fb0ec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2020-09-08T16:47:02.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:47:02.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:02.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:02.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:02.029 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:02.029 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:02.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:02.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:02.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:02.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:02.134 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:02.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:02.135 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:02.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:02.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:02.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:02.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:02.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:02.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:02.138 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:02.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:02.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:02.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:02.141 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:02.142 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:02.142 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:02.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:02.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:02.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:02.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2020-09-08T16:47:02.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2020-09-08T16:47:02.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:02.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:47:02.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:02.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:02.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:02.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:02.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:02.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:02.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:02.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:02.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:02.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:02.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:02.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:02.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:02.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:02.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:02.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:02.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:02.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:02.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:02.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:02.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:02.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:02.152 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:02.152 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:02.152 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:02.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:02.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:02.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:02.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:47:02.155 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:47:02.155 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:02.155 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:02.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:47:02.156 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:47:02.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:47:02.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:47:02.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:47:02.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2020-09-08T16:47:02.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:02.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:47:02.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:47:02.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:47:02.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:47:02.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2020-09-08T16:47:02.448 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:47:02.449 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:02.449 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:02.449 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:02.449 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:02.449 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:02.450 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:02.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=jerasure_sse3 2020-09-08T16:47:02.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:47:02.609 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:47:02.609 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:02.610 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:02.610 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:47:02.610 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:47:02.610 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:02.610 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:02.611 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:02.611 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:02.611 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:02.611 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:02.611 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:02.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:02.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:47:02.697 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:47:02.697 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:02.697 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:02.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:47:02.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:47:02.698 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:02.698 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:02.698 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:02.699 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:02.699 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:02.699 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:02.699 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:02.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:02.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:47:02.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:47:02.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:02.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:47:02.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:47:02.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:47:02.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:47:02.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:47:03.666 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:47:03.666 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:03.666 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:03.667 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:03.667 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:03.667 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:03.667 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:03.668 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:03.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:03.687 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2020-09-08T16:47:03.688 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:03.688 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:03.688 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:03.688 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:03.689 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:03.689 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:03.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:03.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:47:03.769 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:03.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2020-09-08T16:47:03.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:03.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:47:03.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:47:03.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:47:03.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:47:03.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:47:03.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:47:03.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:47:03.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:47:03.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:47:03.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:47:03.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:47:03.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:47:03.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:47:03.782 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:47:03.782 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:03.782 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:03.782 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:03.783 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:03.783 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:03.783 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:03.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:03.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:47:03.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:47:03.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:47:03.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:47:03.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:47:03.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:47:03.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:47:03.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:03.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:47:03.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2020-09-08T16:47:03.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:47:03.786 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:47:03.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3cead3f6-4ebd-401c-8898-482500950e98 2020-09-08T16:47:03.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 3cead3f6-4ebd-401c-8898-482500950e98' 2020-09-08T16:47:03.787 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:47:03.787 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 3cead3f6-4ebd-401c-8898-482500950e98 2020-09-08T16:47:03.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCHtVdfLHKULxAAd92sZSXz0sPOQxB5hkTYBQ== 2020-09-08T16:47:03.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCHtVdfLHKULxAAd92sZSXz0sPOQxB5hkTYBQ=="}' 2020-09-08T16:47:03.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3cead3f6-4ebd-401c-8898-482500950e98 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:04.125 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:04.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:04.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQCHtVdfLHKULxAAd92sZSXz0sPOQxB5hkTYBQ== --osd-uuid 3cead3f6-4ebd-401c-8898-482500950e98 2020-09-08T16:47:04.156 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:04.161+0000 7f4543b96ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:04.158 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:04.163+0000 7f4543b96ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:04.164+0000 7f4543b96ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:04.165+0000 7f4543b96ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:04.165+0000 7f4543b96ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:04.161 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:04.165+0000 7f4543b96ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:04.199 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:04.203+0000 7f4543b96ec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:47:08.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:47:08.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:47:08.556 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:47:08.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:47:08.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:47:09.046 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:47:09.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure_sse3 2020-09-08T16:47:09.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:47:09.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:47:09.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:47:09.048 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:47:09.066 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:09.070+0000 7f40e84baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:09.067 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:09.072+0000 7f40e84baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:09.068 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:09.073+0000 7f40e84baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:09.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:47:09.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:47:09.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:47:09.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:47:09.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:47:09.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:09.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:47:09.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:09.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:09.435 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:09.593 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:09.598+0000 7f40e84baec0 -1 Falling back to public interface 2020-09-08T16:47:09.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:47:10.196 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:10.201+0000 7f40e84baec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:47:10.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:47:10.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:10.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:47:10.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:10.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:10.824 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:11.188 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/29746,v1:127.0.0.1:6803/29746] [v2:127.0.0.1:6804/29746,v1:127.0.0.1:6805/29746] exists,up 3cead3f6-4ebd-401c-8898-482500950e98 2020-09-08T16:47:11.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:47:11.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:47:11.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:47:11.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2020-09-08T16:47:11.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:47:11.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:47:11.190 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:11.190 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:11.190 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:11.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:47:11.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:11.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:47:11.278 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:11.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:47:11.286 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:02.477+0000 7ff2455e76c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2020-09-08T16:47:11.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:47:11.288 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:09.599+0000 7f40e84baec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2020-09-08T16:47:11.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:47:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:11.289 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:11.290 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:11.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:11.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:11.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:11.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:11.394 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:11.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:11.395 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:11.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:11.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:11.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:11.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:11.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:11.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:11.399 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:11.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:11.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:11.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:11.401 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:11.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:11.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:11.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:11.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:11.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:11.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2020-09-08T16:47:11.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2020-09-08T16:47:11.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:11.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:47:11.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:11.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:11.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:11.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:11.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:11.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:11.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:11.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:11.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:11.408 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:11.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:11.409 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:11.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:11.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:11.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:11.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:11.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:11.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:11.412 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:11.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:11.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:11.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:11.413 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:11.413 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:11.414 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:11.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:11.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:11.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:11.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:47:11.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:47:11.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:11.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:11.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:47:11.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:47:11.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:47:11.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:47:11.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:47:11.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2020-09-08T16:47:11.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:11.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:47:11.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:47:11.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:47:11.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:47:11.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2020-09-08T16:47:11.699 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:47:11.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:11.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:11.700 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:11.700 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:11.701 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:11.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:11.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=jerasure_sse4 2020-09-08T16:47:11.884 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:47:11.886 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:47:11.886 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:11.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:11.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:47:11.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:47:11.887 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:11.887 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:11.888 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:11.888 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:11.888 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:11.888 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:11.888 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:11.889 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:11.889 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:47:11.973 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:47:11.973 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:11.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:11.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:47:11.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:47:11.974 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:11.974 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:11.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:11.975 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:11.975 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:11.975 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:11.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:11.976 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:11.976 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:47:12.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:47:12.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:12.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:47:12.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:47:12.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:47:12.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:47:12.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:47:12.984 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:47:12.985 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:12.985 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:12.985 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:12.985 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:12.986 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:12.986 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:12.987 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:12.987 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:13.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2020-09-08T16:47:13.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:13.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:13.008 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:13.008 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:13.008 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:13.008 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:13.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:13.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:47:13.100 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:13.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2020-09-08T16:47:13.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:13.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:47:13.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:47:13.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:47:13.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:47:13.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:47:13.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:47:13.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:47:13.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:47:13.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:47:13.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:47:13.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:47:13.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:47:13.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:47:13.113 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:47:13.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:13.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:13.114 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:13.114 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:13.114 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:13.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:13.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:13.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:47:13.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:47:13.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:47:13.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:47:13.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:47:13.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:47:13.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:47:13.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:13.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:47:13.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2020-09-08T16:47:13.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:47:13.117 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:47:13.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c2355c44-2198-4f4c-83ee-7f3a80288ad6 2020-09-08T16:47:13.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c2355c44-2198-4f4c-83ee-7f3a80288ad6' 2020-09-08T16:47:13.118 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:47:13.119 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 c2355c44-2198-4f4c-83ee-7f3a80288ad6 2020-09-08T16:47:13.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCRtVdfGwWtBxAAsiuuaskC99wlP+N3w3DdsA== 2020-09-08T16:47:13.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCRtVdfGwWtBxAAsiuuaskC99wlP+N3w3DdsA=="}' 2020-09-08T16:47:13.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c2355c44-2198-4f4c-83ee-7f3a80288ad6 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:13.458 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:13.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:13.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQCRtVdfGwWtBxAAsiuuaskC99wlP+N3w3DdsA== --osd-uuid c2355c44-2198-4f4c-83ee-7f3a80288ad6 2020-09-08T16:47:13.489 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:13.494+0000 7f80d526dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:13.491 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:13.495+0000 7f80d526dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:13.492 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:13.497+0000 7f80d526dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:13.493 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:13.498+0000 7f80d526dec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:13.493 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:13.498+0000 7f80d526dec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:13.494 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:13.498+0000 7f80d526dec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:13.541 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:13.538+0000 7f80d526dec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:47:17.897 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:47:17.897 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:47:17.898 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:47:17.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:47:17.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:47:18.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:47:18.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=jerasure_sse4 2020-09-08T16:47:18.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:47:18.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:47:18.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:47:18.395 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:47:18.413 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:18.418+0000 7f7fde7d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:18.438 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:18.443+0000 7f7fde7d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:18.439 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:18.445+0000 7f7fde7d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:18.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:47:18.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:47:18.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:47:18.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:47:18.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:47:18.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:18.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:47:18.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:18.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:18.764 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:18.955 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:18.960+0000 7f7fde7d4ec0 -1 Falling back to public interface 2020-09-08T16:47:19.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:47:19.547 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:19.552+0000 7f7fde7d4ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:47:20.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:47:20.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:20.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:47:20.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:20.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:20.132 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:20.497 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/31014,v1:127.0.0.1:6803/31014] [v2:127.0.0.1:6804/31014,v1:127.0.0.1:6805/31014] exists,up c2355c44-2198-4f4c-83ee-7f3a80288ad6 2020-09-08T16:47:20.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:47:20.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:47:20.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:47:20.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2020-09-08T16:47:20.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:47:20.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:47:20.499 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:20.499 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:20.500 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:20.500 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:47:20.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:20.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:47:20.587 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:20.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:47:20.596 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:11.728+0000 7fc18e7c26c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2020-09-08T16:47:20.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:47:20.597 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:18.961+0000 7f7fde7d4ec0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2020-09-08T16:47:20.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:47:20.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:20.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:20.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:20.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:20.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:20.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:20.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:20.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:20.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:20.713 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:20.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:20.713 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:20.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:20.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:20.714 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:20.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:20.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:20.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:20.715 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:20.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:20.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:20.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:20.715 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:20.716 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:20.716 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:20.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:20.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:20.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:20.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2020-09-08T16:47:20.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2020-09-08T16:47:20.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:20.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:47:20.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:20.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:20.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:20.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:20.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:20.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:20.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:20.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:20.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:20.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:20.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:20.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:20.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:20.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:20.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:20.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:20.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:20.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:20.723 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:20.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:20.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:20.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:20.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:20.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:20.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:20.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:20.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:20.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:20.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:47:20.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:47:20.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:20.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:20.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:47:20.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:47:20.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:47:20.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:47:20.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:47:20.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2020-09-08T16:47:20.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:20.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:47:20.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:47:20.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:47:20.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:47:20.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2020-09-08T16:47:20.960 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:47:20.960 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:20.961 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:20.961 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:20.961 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:20.961 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:20.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:20.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=shec_generic 2020-09-08T16:47:21.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:47:21.162 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:47:21.162 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:21.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:21.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:47:21.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:47:21.163 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:21.163 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:21.164 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:21.164 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:21.164 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:21.164 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:21.164 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:21.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:21.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:47:21.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:47:21.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:21.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:21.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:47:21.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:47:21.248 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:21.248 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:21.248 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:21.249 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:21.249 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:21.249 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:21.249 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:21.249 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:21.250 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:47:21.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:47:21.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:21.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:47:21.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:47:21.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:47:21.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:47:21.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:47:22.209 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:47:22.210 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:22.210 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:22.210 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:22.210 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:22.211 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:22.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:22.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:22.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:22.231 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2020-09-08T16:47:22.232 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:22.232 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:22.232 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:22.232 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:22.233 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:22.233 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:22.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:22.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:47:22.313 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:22.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2020-09-08T16:47:22.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:22.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:47:22.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:47:22.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:47:22.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:47:22.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:47:22.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:47:22.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:47:22.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:47:22.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:47:22.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:47:22.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:47:22.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:47:22.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:47:22.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:47:22.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:22.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:22.345 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:22.345 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:22.345 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:22.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:22.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:22.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:47:22.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:47:22.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:47:22.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:47:22.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:47:22.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:47:22.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:47:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:47:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2020-09-08T16:47:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:47:22.349 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:47:22.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2b00ad9d-2d7b-4481-b324-3a18c132e09c 2020-09-08T16:47:22.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 2b00ad9d-2d7b-4481-b324-3a18c132e09c' 2020-09-08T16:47:22.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:47:22.350 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 2b00ad9d-2d7b-4481-b324-3a18c132e09c 2020-09-08T16:47:22.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCatVdfwY1rFBAAPfexppO4rMIoXnuzvzvUNA== 2020-09-08T16:47:22.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCatVdfwY1rFBAAPfexppO4rMIoXnuzvzvUNA=="}' 2020-09-08T16:47:22.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2b00ad9d-2d7b-4481-b324-3a18c132e09c -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:22.708 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:22.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:22.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec_generic --mkfs --key AQCatVdfwY1rFBAAPfexppO4rMIoXnuzvzvUNA== --osd-uuid 2b00ad9d-2d7b-4481-b324-3a18c132e09c 2020-09-08T16:47:22.740 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:22.745+0000 7f9fc3de1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:22.746 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:22.747+0000 7f9fc3de1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:22.746 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:22.749+0000 7f9fc3de1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:22.747 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:22.749+0000 7f9fc3de1ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:22.747 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:22.750+0000 7f9fc3de1ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:22.747 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:22.750+0000 7f9fc3de1ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:22.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:22.789+0000 7f9fc3de1ec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:47:27.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:47:27.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:47:27.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:47:27.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:47:27.112 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:47:27.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:47:27.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:47:27.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec_generic 2020-09-08T16:47:27.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:47:27.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:47:27.608 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:47:27.626 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:27.632+0000 7f26ebab1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:27.628 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:27.634+0000 7f26ebab1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:27.629 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:27.635+0000 7f26ebab1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:28.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:47:28.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:47:28.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:47:28.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:47:28.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:47:28.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:28.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:47:28.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:28.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:28.029 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:28.145 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:28.151+0000 7f26ebab1ec0 -1 Falling back to public interface 2020-09-08T16:47:28.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:47:28.980 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:28.986+0000 7f26ebab1ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:47:29.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:47:29.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:29.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:47:29.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:29.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:29.394 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:29.775 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/32282,v1:127.0.0.1:6803/32282] [v2:127.0.0.1:6804/32282,v1:127.0.0.1:6805/32282] exists,up 2b00ad9d-2d7b-4481-b324-3a18c132e09c 2020-09-08T16:47:29.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:47:29.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:47:29.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:47:29.776 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2020-09-08T16:47:29.776 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:47:29.777 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:47:29.777 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:29.777 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:29.777 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:29.777 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:47:29.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:29.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:47:29.864 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:29.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:47:29.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:47:29.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:47:29.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:29.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:29.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:29.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:29.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:29.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:29.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:29.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:29.893 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:20.990+0000 7f3a74abe6c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2020-09-08T16:47:29.893 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:28.152+0000 7f26ebab1ec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2020-09-08T16:47:29.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:29.981 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:29.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:29.982 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:29.982 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:29.982 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:29.983 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:29.984 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:29.984 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:29.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:29.985 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:29.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:29.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:29.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:29.988 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:29.988 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:29.988 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:29.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:29.990 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:29.990 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:29.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2020-09-08T16:47:29.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2020-09-08T16:47:29.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:29.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:47:29.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:29.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:29.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:29.992 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:29.992 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:29.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:29.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:29.993 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:29.994 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:29.994 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:29.994 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:29.994 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:29.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:29.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:29.995 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:29.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:29.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:29.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:29.996 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:29.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:29.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:29.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:29.998 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:29.998 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:29.998 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:29.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:30.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:30.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:30.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:47:30.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:47:30.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:30.001 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:30.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:47:30.002 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:47:30.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:47:30.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:47:30.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:47:30.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2020-09-08T16:47:30.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:30.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:47:30.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:47:30.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:47:30.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:47:30.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2020-09-08T16:47:30.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:47:30.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:30.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:30.300 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:30.300 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:30.300 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:30.300 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:30.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=shec_sse3 2020-09-08T16:47:30.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:47:30.476 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:47:30.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:30.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:30.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:47:30.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:47:30.477 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:30.478 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:30.478 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:30.478 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:30.478 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:30.478 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:30.479 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:30.479 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:30.479 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:47:30.564 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:47:30.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:30.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:30.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:47:30.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:47:30.566 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:30.566 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:30.566 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:30.566 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:30.566 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:30.567 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:30.567 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:30.567 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:30.567 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:47:30.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:47:30.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:30.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:47:30.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:47:30.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:47:30.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:47:30.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:47:31.525 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:47:31.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:31.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:31.526 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:31.526 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:31.527 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:31.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:31.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:31.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:31.547 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2020-09-08T16:47:31.547 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:31.547 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:31.548 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:31.548 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:31.548 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:31.548 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:31.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:31.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:47:31.628 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:31.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2020-09-08T16:47:31.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:31.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:47:31.640 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:47:31.640 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:47:31.640 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:47:31.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:47:31.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:47:31.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:47:31.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:47:31.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:47:31.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:47:31.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:47:31.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:47:31.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:47:31.643 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:47:31.643 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:31.643 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:31.643 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:31.643 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:31.644 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:31.644 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:31.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:31.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:47:31.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:47:31.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:47:31.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:47:31.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:47:31.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:47:31.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:47:31.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:31.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:47:31.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2020-09-08T16:47:31.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:47:31.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:47:31.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b19e672c-ef53-441b-bd8c-e5fca6a3f745 2020-09-08T16:47:31.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 b19e672c-ef53-441b-bd8c-e5fca6a3f745' 2020-09-08T16:47:31.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:47:31.647 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 b19e672c-ef53-441b-bd8c-e5fca6a3f745 2020-09-08T16:47:31.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCjtVdf0WwqJxAAXpQhdcmO+r2E1hv+JT6y6w== 2020-09-08T16:47:31.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCjtVdf0WwqJxAAXpQhdcmO+r2E1hv+JT6y6w=="}' 2020-09-08T16:47:31.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b19e672c-ef53-441b-bd8c-e5fca6a3f745 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:31.966 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:31.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:31.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQCjtVdf0WwqJxAAXpQhdcmO+r2E1hv+JT6y6w== --osd-uuid b19e672c-ef53-441b-bd8c-e5fca6a3f745 2020-09-08T16:47:31.998 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:32.002+0000 7f3c306faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:32.000 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:32.004+0000 7f3c306faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:32.001 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:32.005+0000 7f3c306faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:32.002 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:32.007+0000 7f3c306faec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:32.003 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:32.007+0000 7f3c306faec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:32.003 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:32.007+0000 7f3c306faec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:32.033 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:32.037+0000 7f3c306faec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:47:36.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:47:36.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:47:36.403 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:47:36.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:47:36.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:47:36.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:47:36.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec_sse3 2020-09-08T16:47:36.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:47:36.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:47:36.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:47:36.896 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:47:36.914 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:36.918+0000 7fdc49b77ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:36.915 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:36.920+0000 7fdc49b77ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:36.917 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:36.922+0000 7fdc49b77ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:37.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:47:37.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:47:37.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:47:37.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:47:37.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:47:37.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:37.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:47:37.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:37.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:37.363 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:37.434 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:37.438+0000 7fdc49b77ec0 -1 Falling back to public interface 2020-09-08T16:47:37.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:47:38.263 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:38.268+0000 7fdc49b77ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:47:38.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:47:38.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:38.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:47:38.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:38.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:38.729 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:39.093 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/33558,v1:127.0.0.1:6803/33558] [v2:127.0.0.1:6804/33558,v1:127.0.0.1:6805/33558] exists,up b19e672c-ef53-441b-bd8c-e5fca6a3f745 2020-09-08T16:47:39.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:47:39.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:47:39.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:47:39.094 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2020-09-08T16:47:39.094 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:47:39.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:47:39.095 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:39.095 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:39.095 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:39.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:47:39.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:39.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:47:39.182 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:39.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:47:39.191 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:30.329+0000 7f95c390c6c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2020-09-08T16:47:39.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:47:39.192 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:37.439+0000 7fdc49b77ec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2020-09-08T16:47:39.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:47:39.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:39.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:39.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:39.194 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:39.194 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:39.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:39.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:39.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:39.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:39.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:39.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:39.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:39.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:39.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:39.308 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:39.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:39.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:39.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:39.312 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:39.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:39.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:39.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:39.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:39.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:39.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:39.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:39.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:39.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:39.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2020-09-08T16:47:39.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2020-09-08T16:47:39.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:39.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2020-09-08T16:47:39.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:39.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:39.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:39.315 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:39.316 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:39.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:39.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:39.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:39.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:39.317 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:39.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:39.317 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:39.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:39.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:39.317 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:39.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:39.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:39.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:39.321 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:39.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:39.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:39.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:39.322 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:39.322 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:39.322 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:39.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:39.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:39.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:39.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2020-09-08T16:47:39.323 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:47:39.323 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:39.323 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:39.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17314 2020-09-08T16:47:39.324 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:47:39.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:47:39.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:47:39.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2020-09-08T16:47:39.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2020-09-08T16:47:39.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:39.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:47:39.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:47:39.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:47:39.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2020-09-08T16:47:39.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2020-09-08T16:47:39.649 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:47:39.649 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:39.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:39.650 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:39.650 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:39.650 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:39.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:39.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_erasure_code_plugins=shec_sse4 2020-09-08T16:47:39.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:47:39.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:47:39.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:39.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:39.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:47:39.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:47:39.870 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:39.870 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:39.870 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:39.871 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:39.871 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:39.871 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:39.871 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:39.871 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:39.872 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get fsid 2020-09-08T16:47:39.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:47:39.956 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:39.956 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:39.956 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:47:39.956 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:47:39.957 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:39.957 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:39.957 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:39.957 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:39.957 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:39.958 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:39.958 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:39.958 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:39.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.17314/ceph-mon.a.asok config get mon_host 2020-09-08T16:47:40.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2020-09-08T16:47:40.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:40.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:47:40.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:47:40.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:47:40.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code-plugins/x 2020-09-08T16:47:40.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:47:40.917 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:47:40.918 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:40.918 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:40.918 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:40.918 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:40.918 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:40.919 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:40.919 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:40.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:40.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2020-09-08T16:47:40.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:40.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:40.940 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:40.940 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:40.940 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:40.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-mon.a.asok 2020-09-08T16:47:40.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:40.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-mon.a.asok log flush 2020-09-08T16:47:41.021 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:41.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2020-09-08T16:47:41.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:41.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:47:41.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:47:41.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:47:41.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code-plugins/0 2020-09-08T16:47:41.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 ' 2020-09-08T16:47:41.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:47:41.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:47:41.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:47:41.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2020-09-08T16:47:41.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2020-09-08T16:47:41.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:47:41.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:47:41.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2020-09-08T16:47:41.035 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:47:41.035 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:41.036 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:41.036 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:41.036 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:41.036 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:41.036 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:41.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' 2020-09-08T16:47:41.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:47:41.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:47:41.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:47:41.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2020-09-08T16:47:41.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2020-09-08T16:47:41.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:47:41.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:47:41.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:41.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:47:41.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2020-09-08T16:47:41.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code-plugins/0 2020-09-08T16:47:41.039 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:47:41.040 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=13e5f75a-47ec-4cfd-8eae-4c3b10e65976 2020-09-08T16:47:41.040 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 13e5f75a-47ec-4cfd-8eae-4c3b10e65976' 2020-09-08T16:47:41.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:47:41.040 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 13e5f75a-47ec-4cfd-8eae-4c3b10e65976 2020-09-08T16:47:41.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCttVdfOhPwAhAAziJGayw0ZSOSj12CIL732w== 2020-09-08T16:47:41.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCttVdfOhPwAhAAziJGayw0ZSOSj12CIL732w=="}' 2020-09-08T16:47:41.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 13e5f75a-47ec-4cfd-8eae-4c3b10e65976 -i td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:41.393 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:41.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code-plugins/0/new.json 2020-09-08T16:47:41.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQCttVdfOhPwAhAAziJGayw0ZSOSj12CIL732w== --osd-uuid 13e5f75a-47ec-4cfd-8eae-4c3b10e65976 2020-09-08T16:47:41.427 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:41.429+0000 7f5e38331ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:41.428 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:41.431+0000 7f5e38331ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:41.428 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:41.432+0000 7f5e38331ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:41.429 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:41.432+0000 7f5e38331ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:41.429 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:41.432+0000 7f5e38331ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:41.429 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:41.432+0000 7f5e38331ec0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2020-09-08T16:47:41.467 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:41.471+0000 7f5e38331ec0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2020-09-08T16:47:45.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2020-09-08T16:47:45.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:47:45.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:47:45.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:47:45.917 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:47:46.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:47:46.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:47:46.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:47:46.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:47:46.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4999949c-0304-4378-9d08-e8a7dc084e1c --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17314/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd_erasure_code_plugins=shec_sse4 2020-09-08T16:47:46.413 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:47:46.432 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:46.435+0000 7f69dbb6fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:46.432 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:46.437+0000 7f69dbb6fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:46.433 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:46.438+0000 7f69dbb6fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:46.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:47:46.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:47:46.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:47:46.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:47:46.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:47:46.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:46.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:47:46.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:46.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:46.781 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:46.950 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:46.954+0000 7f69dbb6fec0 -1 Falling back to public interface 2020-09-08T16:47:47.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:47:47.764 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:47.768+0000 7f69dbb6fec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:47:48.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:47:48.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:48.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:47:48.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:48.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:48.168 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:48.534 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34826,v1:127.0.0.1:6803/34826] [v2:127.0.0.1:6804/34826,v1:127.0.0.1:6805/34826] exists,up 13e5f75a-47ec-4cfd-8eae-4c3b10e65976 2020-09-08T16:47:48.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:47:48.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:47:48.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:47:48.535 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2020-09-08T16:47:48.536 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:47:48.536 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:47:48.537 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:48.537 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:48.537 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:48.537 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17314/ceph-osd.0.asok 2020-09-08T16:47:48.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2020-09-08T16:47:48.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17314/ceph-osd.0.asok log flush 2020-09-08T16:47:48.624 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:48.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2020-09-08T16:47:48.632 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:39.678+0000 7f71d83196c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2020-09-08T16:47:48.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2020-09-08T16:47:48.633 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:46.955+0000 7f69dbb6fec0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2020-09-08T16:47:48.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2020-09-08T16:47:48.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:48.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:48.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:48.634 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:48.635 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:48.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:48.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:48.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:48.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:48.739 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:48.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:48.740 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:48.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:48.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:48.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:48.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:48.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:48.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:48.744 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:48.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:48.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:48.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:48.747 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:48.747 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:48.747 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:48.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:48.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:48.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:48.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2020-09-08T16:47:48.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-09-08T16:47:48.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/test-erasure-code-plugins 0 2020-09-08T16:47:48.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2020-09-08T16:47:48.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-09-08T16:47:48.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2020-09-08T16:47:48.751 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:48.751 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:48.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:48.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:48.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:48.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:48.753 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:48.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:48.753 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:48.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:48.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:48.754 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:48.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:48.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:48.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:48.755 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:48.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:48.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-09-08T16:47:48.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2020-09-08T16:47:48.757 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:48.757 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:48.757 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17314 2020-09-08T16:47:48.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17314 2020-09-08T16:47:48.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:48.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:48.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-09-08T16:47:48.760 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-09-08T16:47:48.761 INFO:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-09-08T16:47:48.789 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2020-09-08T16:47:48.790 INFO:teuthology.orchestra.run.smithi120:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dce4d291bd4243c9802c17d67b85a6da6bbc0f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2020-09-08T16:47:48.849 INFO:tasks.workunit.client.0.smithi120.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-09-08T16:47:48.858 INFO:tasks.workunit.client.0.smithi120.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-09-08T16:47:48.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-09-08T16:47:48.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-09-08T16:47:48.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-09-08T16:47:48.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-09-08T16:47:48.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-09-08T16:47:48.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-09-08T16:47:48.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-09-08T16:47:48.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-09-08T16:47:48.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/test-erasure-code 2020-09-08T16:47:48.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2020-09-08T16:47:48.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2020-09-08T16:47:48.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2020-09-08T16:47:48.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2020-09-08T16:47:48.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2020-09-08T16:47:48.862 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2020-09-08T16:47:48.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none ' 2020-09-08T16:47:48.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:47:48.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2020-09-08T16:47:48.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2020-09-08T16:47:48.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2020-09-08T16:47:48.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2020-09-08T16:47:48.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:47:48.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2020-09-08T16:47:48.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:47:48.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:47:48.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:47:48.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:47:48.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:47:48.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:47:48.868 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:47:48.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:47:48.868 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:47:48.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:47:48.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:47:48.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:47:48.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:48.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:47:48.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:47:48.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:47:48.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:47:48.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:47:48.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2020-09-08T16:47:48.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:47:48.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:48.871 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:48.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35290 2020-09-08T16:47:48.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:47:48.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:47:48.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2020-09-08T16:47:48.872 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:47:48.872 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:48.872 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:48.872 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.35290 2020-09-08T16:47:48.873 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:47:48.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-09-08T16:47:48.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-09-08T16:47:48.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:47:48.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2020-09-08T16:47:48.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2020-09-08T16:47:48.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2020-09-08T16:47:48.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:47:48.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:47:48.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:47:48.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2020-09-08T16:47:48.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2020-09-08T16:47:49.175 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:47:49.175 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:49.176 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:49.176 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:49.176 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:49.176 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:49.176 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:47:49.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T16:47:49.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:47:49.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:47:49.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:49.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:49.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:47:49.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:47:49.347 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:49.347 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:49.347 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:49.347 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:49.348 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:49.348 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:49.348 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35290/ceph-mon.a.asok 2020-09-08T16:47:49.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:49.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.35290/ceph-mon.a.asok config get fsid 2020-09-08T16:47:49.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:47:49.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:47:49.434 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:47:49.434 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:47:49.434 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:47:49.435 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:47:49.435 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:49.435 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:49.436 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:49.436 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:49.436 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:49.437 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35290/ceph-mon.a.asok 2020-09-08T16:47:49.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:47:49.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.35290/ceph-mon.a.asok config get mon_host 2020-09-08T16:47:49.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2020-09-08T16:47:49.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-code 2020-09-08T16:47:49.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:47:49.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:47:49.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:47:49.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-code/x 2020-09-08T16:47:49.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:47:50.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:47:50.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:50.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:50.406 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:50.406 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:50.406 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:50.407 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:47:50.407 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:50.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:47:50.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2020-09-08T16:47:50.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:47:50.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:47:50.425 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:47:50.426 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:50.426 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:50.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35290/ceph-mon.a.asok 2020-09-08T16:47:50.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2020-09-08T16:47:50.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.35290/ceph-mon.a.asok log flush 2020-09-08T16:47:50.505 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:47:50.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2020-09-08T16:47:50.519 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:49.209+0000 7f77a0ade6c0 0 load: jerasure load: lrc load: isa 2020-09-08T16:47:50.519 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 274522d6-5eaf-4aae-947e-f3a084ba5954 2020-09-08T16:47:50.519 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2020-09-08T16:47:50.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:47:50.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2020-09-08T16:47:50.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:47:50.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:47:50.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:47:50.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:47:50.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/0 2020-09-08T16:47:50.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:47:50.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:47:50.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:47:50.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:47:50.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2020-09-08T16:47:50.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2020-09-08T16:47:50.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:47:50.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:47:50.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:47:50.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:47:50.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:50.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:50.527 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:50.527 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:50.527 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:50.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:47:50.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:47:50.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:47:50.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:47:50.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:47:50.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:47:50.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:47:50.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:47:50.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:47:50.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:50.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:47:50.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:47:50.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/0 2020-09-08T16:47:50.530 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:47:50.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=274522d6-5eaf-4aae-947e-f3a084ba5954 2020-09-08T16:47:50.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 274522d6-5eaf-4aae-947e-f3a084ba5954' 2020-09-08T16:47:50.530 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:47:50.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC2tVdfuNz6HxAA5qJj7rVi/tAjpIv+i57ZLQ== 2020-09-08T16:47:50.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC2tVdfuNz6HxAA5qJj7rVi/tAjpIv+i57ZLQ=="}' 2020-09-08T16:47:50.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 274522d6-5eaf-4aae-947e-f3a084ba5954 -i td/test-erasure-code/0/new.json 2020-09-08T16:47:50.862 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:50.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/0/new.json 2020-09-08T16:47:50.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC2tVdfuNz6HxAA5qJj7rVi/tAjpIv+i57ZLQ== --osd-uuid 274522d6-5eaf-4aae-947e-f3a084ba5954 2020-09-08T16:47:50.894 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:50.898+0000 7f8b06120ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:50.895 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:50.900+0000 7f8b06120ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:50.896 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:50.901+0000 7f8b06120ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:50.897 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:50.901+0000 7f8b06120ec0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2020-09-08T16:47:50.897 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:50.901+0000 7f8b06120ec0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2020-09-08T16:47:50.897 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:50.901+0000 7f8b06120ec0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2020-09-08T16:47:50.942 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:50.947+0000 7f8b06120ec0 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2020-09-08T16:47:55.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/0/keyring 2020-09-08T16:47:55.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:47:55.508 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:47:55.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:47:55.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:47:56.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:47:56.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:56.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:47:56.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:47:56.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:47:56.003 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:47:56.021 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:56.026+0000 7f99463adec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:56.023 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:56.028+0000 7f99463adec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:56.025 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:56.029+0000 7f99463adec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:56.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:47:56.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:47:56.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:47:56.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:47:56.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:47:56.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:56.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:47:56.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:56.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:56.371 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:47:56.546 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:56.549+0000 7f99463adec0 -1 Falling back to public interface 2020-09-08T16:47:56.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:47:57.365 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:57.370+0000 7f99463adec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:47:57.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:47:57.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:47:57.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:47:57.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:47:57.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:47:57.771 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:58.162 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/36151,v1:127.0.0.1:6803/36151] [v2:127.0.0.1:6804/36151,v1:127.0.0.1:6805/36151] exists,up 274522d6-5eaf-4aae-947e-f3a084ba5954 2020-09-08T16:47:58.162 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 8d55d0ed-0dbe-4b95-b638-cad02d013bea 2020-09-08T16:47:58.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:47:58.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:47:58.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:47:58.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:47:58.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2020-09-08T16:47:58.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:47:58.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:47:58.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T16:47:58.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:47:58.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/1 2020-09-08T16:47:58.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:47:58.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:47:58.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:47:58.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:47:58.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2020-09-08T16:47:58.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2020-09-08T16:47:58.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:47:58.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:47:58.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:47:58.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:47:58.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:47:58.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:47:58.167 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:47:58.167 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:47:58.167 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:47:58.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:47:58.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:47:58.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:47:58.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:47:58.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:47:58.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:47:58.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:47:58.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:47:58.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:47:58.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:47:58.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:47:58.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:47:58.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/1 2020-09-08T16:47:58.180 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:47:58.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8d55d0ed-0dbe-4b95-b638-cad02d013bea 2020-09-08T16:47:58.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 8d55d0ed-0dbe-4b95-b638-cad02d013bea' 2020-09-08T16:47:58.181 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:47:58.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC+tVdf4H/AChAAgrNNn3cbfarEvfb9B3IeiA== 2020-09-08T16:47:58.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC+tVdf4H/AChAAgrNNn3cbfarEvfb9B3IeiA=="}' 2020-09-08T16:47:58.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8d55d0ed-0dbe-4b95-b638-cad02d013bea -i td/test-erasure-code/1/new.json 2020-09-08T16:47:58.813 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:47:58.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/1/new.json 2020-09-08T16:47:58.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC+tVdf4H/AChAAgrNNn3cbfarEvfb9B3IeiA== --osd-uuid 8d55d0ed-0dbe-4b95-b638-cad02d013bea 2020-09-08T16:47:58.845 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:58.850+0000 7f156fa24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:58.847 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:58.852+0000 7f156fa24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:58.848 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:58.852+0000 7f156fa24ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:47:58.848 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:58.853+0000 7f156fa24ec0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2020-09-08T16:47:58.848 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:58.853+0000 7f156fa24ec0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2020-09-08T16:47:58.849 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:58.853+0000 7f156fa24ec0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2020-09-08T16:47:58.963 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:47:58.968+0000 7f156fa24ec0 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2020-09-08T16:48:03.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/1/keyring 2020-09-08T16:48:03.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:03.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T16:48:03.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:03.587 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T16:48:04.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T16:48:04.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:04.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:48:04.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:48:04.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:48:04.088 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T16:48:04.106 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:04.112+0000 7f18e5233ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:04.108 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:04.114+0000 7f18e5233ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:04.109 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:04.115+0000 7f18e5233ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:04.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T16:48:04.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:48:04.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T16:48:04.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:48:04.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:48:04.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:04.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:48:04.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:04.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:48:04.467 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:48:04.627 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:04.632+0000 7f18e5233ec0 -1 Falling back to public interface 2020-09-08T16:48:04.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:48:05.455 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:05.461+0000 7f18e5233ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T16:48:05.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:48:05.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:05.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:48:05.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:05.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:48:05.831 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:48:06.239 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/37145,v1:127.0.0.1:6811/37145] [v2:127.0.0.1:6812/37145,v1:127.0.0.1:6813/37145] exists,up 8d55d0ed-0dbe-4b95-b638-cad02d013bea 2020-09-08T16:48:06.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:48:06.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:48:06.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:48:06.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:48:06.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2020-09-08T16:48:06.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:48:06.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:48:06.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T16:48:06.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:48:06.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/2 2020-09-08T16:48:06.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:48:06.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:48:06.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:48:06.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:48:06.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2020-09-08T16:48:06.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2020-09-08T16:48:06.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:48:06.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:48:06.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:48:06.269 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:48:06.269 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:48:06.270 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:48:06.270 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:48:06.270 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:48:06.270 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:48:06.270 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:06.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:06.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:48:06.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:48:06.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:48:06.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:48:06.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:48:06.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:48:06.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:48:06.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:06.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:48:06.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:48:06.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/2 2020-09-08T16:48:06.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:48:06.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=36b0978d-31cb-44be-8178-818455b56139 2020-09-08T16:48:06.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 36b0978d-31cb-44be-8178-818455b56139' 2020-09-08T16:48:06.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:48:06.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDGtVdfhyl+DxAAbf3R6TQkVrcL0VDDCIlohQ== 2020-09-08T16:48:06.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDGtVdfhyl+DxAAbf3R6TQkVrcL0VDDCIlohQ=="}' 2020-09-08T16:48:06.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 36b0978d-31cb-44be-8178-818455b56139 -i td/test-erasure-code/2/new.json 2020-09-08T16:48:06.276 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 36b0978d-31cb-44be-8178-818455b56139 2020-09-08T16:48:06.740 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:48:06.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/2/new.json 2020-09-08T16:48:06.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGtVdfhyl+DxAAbf3R6TQkVrcL0VDDCIlohQ== --osd-uuid 36b0978d-31cb-44be-8178-818455b56139 2020-09-08T16:48:06.773 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:06.779+0000 7f1fccd76ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:06.775 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:06.780+0000 7f1fccd76ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:06.776 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:06.782+0000 7f1fccd76ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:06.777 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:06.782+0000 7f1fccd76ec0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2020-09-08T16:48:06.777 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:06.782+0000 7f1fccd76ec0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2020-09-08T16:48:06.777 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:06.782+0000 7f1fccd76ec0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2020-09-08T16:48:06.812 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:06.818+0000 7f1fccd76ec0 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2020-09-08T16:48:11.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/2/keyring 2020-09-08T16:48:11.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:11.419 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T16:48:11.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T16:48:11.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:11.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T16:48:11.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:48:11.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:11.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:48:11.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:48:11.918 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T16:48:11.936 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:11.942+0000 7f55541faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:11.938 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:11.943+0000 7f55541faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:11.939 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:11.945+0000 7f55541faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:12.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T16:48:12.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:48:12.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T16:48:12.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:48:12.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:48:12.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:12.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:48:12.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:12.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:48:12.286 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:48:12.455 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:12.460+0000 7f55541faec0 -1 Falling back to public interface 2020-09-08T16:48:12.673 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:48:13.287 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:13.293+0000 7f55541faec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T16:48:13.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:48:13.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:13.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:48:13.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:13.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:48:13.675 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:48:14.055 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/38139,v1:127.0.0.1:6819/38139] [v2:127.0.0.1:6820/38139,v1:127.0.0.1:6821/38139] exists,up 36b0978d-31cb-44be-8178-818455b56139 2020-09-08T16:48:14.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:48:14.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:48:14.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:48:14.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:48:14.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2020-09-08T16:48:14.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:48:14.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:48:14.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T16:48:14.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:48:14.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/3 2020-09-08T16:48:14.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:48:14.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:48:14.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:48:14.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:48:14.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2020-09-08T16:48:14.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2020-09-08T16:48:14.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:48:14.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:48:14.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:48:14.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:48:14.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:48:14.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:48:14.060 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:48:14.060 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:48:14.060 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:48:14.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:14.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:14.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:48:14.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:48:14.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:48:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:48:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:48:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:48:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:48:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:48:14.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:48:14.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/3 2020-09-08T16:48:14.064 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:48:14.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5485caeb-d972-465c-a83e-9237ab4141b7 2020-09-08T16:48:14.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 5485caeb-d972-465c-a83e-9237ab4141b7' 2020-09-08T16:48:14.064 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:48:14.065 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 5485caeb-d972-465c-a83e-9237ab4141b7 2020-09-08T16:48:14.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDOtVdfDU2JBBAAViv5jrktaPfayjhQNWF/Kg== 2020-09-08T16:48:14.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDOtVdfDU2JBBAAViv5jrktaPfayjhQNWF/Kg=="}' 2020-09-08T16:48:14.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5485caeb-d972-465c-a83e-9237ab4141b7 -i td/test-erasure-code/3/new.json 2020-09-08T16:48:14.763 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T16:48:14.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/3/new.json 2020-09-08T16:48:14.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDOtVdfDU2JBBAAViv5jrktaPfayjhQNWF/Kg== --osd-uuid 5485caeb-d972-465c-a83e-9237ab4141b7 2020-09-08T16:48:14.795 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:14.800+0000 7f894c6baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:14.797 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:14.802+0000 7f894c6baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:14.798 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:14.803+0000 7f894c6baec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:14.799 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:14.804+0000 7f894c6baec0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2020-09-08T16:48:14.799 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:14.804+0000 7f894c6baec0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2020-09-08T16:48:14.800 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:14.804+0000 7f894c6baec0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2020-09-08T16:48:14.880 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:14.885+0000 7f894c6baec0 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2020-09-08T16:48:19.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/3/keyring 2020-09-08T16:48:19.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:19.190 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T16:48:19.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T16:48:19.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:19.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T16:48:19.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:48:19.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:48:19.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:48:19.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:19.692 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T16:48:19.704 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:19.707+0000 7f6481cdcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:19.705 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:19.709+0000 7f6481cdcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:19.705 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:19.710+0000 7f6481cdcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T16:48:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:48:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T16:48:20.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:48:20.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:48:20.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:20.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:48:20.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:20.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:48:20.061 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:48:20.221 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:20.226+0000 7f6481cdcec0 -1 Falling back to public interface 2020-09-08T16:48:20.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:48:21.024 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:21.028+0000 7f6481cdcec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T16:48:21.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:48:21.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:21.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:48:21.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:21.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:48:21.433 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:48:21.813 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/39133,v1:127.0.0.1:6827/39133] [v2:127.0.0.1:6828/39133,v1:127.0.0.1:6829/39133] exists,up 5485caeb-d972-465c-a83e-9237ab4141b7 2020-09-08T16:48:21.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:48:21.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:48:21.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:48:21.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:48:21.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2020-09-08T16:48:21.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:48:21.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:48:21.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T16:48:21.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:48:21.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/4 2020-09-08T16:48:21.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:48:21.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:48:21.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:48:21.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:48:21.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2020-09-08T16:48:21.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2020-09-08T16:48:21.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:48:21.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:48:21.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:48:21.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:48:21.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:48:21.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:48:21.818 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:48:21.818 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:48:21.818 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:48:21.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:21.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:21.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:48:21.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:48:21.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:48:21.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:48:21.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:48:21.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:48:21.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:48:21.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:21.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:48:21.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:48:21.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/4 2020-09-08T16:48:21.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:48:21.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c5b6638a-0b77-47f6-9699-3b71ca2ecfcb 2020-09-08T16:48:21.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 c5b6638a-0b77-47f6-9699-3b71ca2ecfcb' 2020-09-08T16:48:21.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:48:21.823 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 c5b6638a-0b77-47f6-9699-3b71ca2ecfcb 2020-09-08T16:48:21.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDVtVdfgIC4MRAAFRIFgv8OzdO0TKi8/YI7gQ== 2020-09-08T16:48:21.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDVtVdfgIC4MRAAFRIFgv8OzdO0TKi8/YI7gQ=="}' 2020-09-08T16:48:21.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c5b6638a-0b77-47f6-9699-3b71ca2ecfcb -i td/test-erasure-code/4/new.json 2020-09-08T16:48:22.281 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T16:48:22.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/4/new.json 2020-09-08T16:48:22.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDVtVdfgIC4MRAAFRIFgv8OzdO0TKi8/YI7gQ== --osd-uuid c5b6638a-0b77-47f6-9699-3b71ca2ecfcb 2020-09-08T16:48:22.314 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:22.318+0000 7fca0db42ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:22.315 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:22.320+0000 7fca0db42ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:22.316 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:22.321+0000 7fca0db42ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:22.317 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:22.321+0000 7fca0db42ec0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2020-09-08T16:48:22.317 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:22.321+0000 7fca0db42ec0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2020-09-08T16:48:22.317 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:22.321+0000 7fca0db42ec0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2020-09-08T16:48:22.365 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:22.370+0000 7fca0db42ec0 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2020-09-08T16:48:26.910 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/4/keyring 2020-09-08T16:48:26.911 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:26.911 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T16:48:26.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:26.912 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T16:48:27.404 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T16:48:27.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T16:48:27.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:27.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:48:27.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:48:27.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:48:27.424 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:27.428+0000 7fc472ebeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:27.426 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:27.430+0000 7fc472ebeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:27.427 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:27.432+0000 7fc472ebeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:27.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T16:48:27.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:48:27.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T16:48:27.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:48:27.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:48:27.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:27.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:48:27.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:27.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:48:27.791 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:48:27.943 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:27.948+0000 7fc472ebeec0 -1 Falling back to public interface 2020-09-08T16:48:28.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:48:28.821 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:28.825+0000 7fc472ebeec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T16:48:29.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:48:29.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:29.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:48:29.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:29.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:48:29.175 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:48:29.641 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/40127,v1:127.0.0.1:6835/40127] [v2:127.0.0.1:6836/40127,v1:127.0.0.1:6837/40127] exists,up c5b6638a-0b77-47f6-9699-3b71ca2ecfcb 2020-09-08T16:48:29.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:48:29.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:48:29.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:48:29.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:48:29.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2020-09-08T16:48:29.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:48:29.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:48:29.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-09-08T16:48:29.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:48:29.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/5 2020-09-08T16:48:29.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:48:29.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:48:29.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:48:29.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:48:29.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2020-09-08T16:48:29.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2020-09-08T16:48:29.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:48:29.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:48:29.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:48:29.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:48:29.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:48:29.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:48:29.647 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:48:29.647 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:48:29.647 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:48:29.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:29.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:29.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:48:29.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:48:29.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:48:29.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:48:29.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:48:29.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:48:29.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:48:29.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:29.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:48:29.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:48:29.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/5 2020-09-08T16:48:29.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:48:29.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3ae8baa2-f695-4ec7-942c-9953e01e86a1 2020-09-08T16:48:29.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 3ae8baa2-f695-4ec7-942c-9953e01e86a1' 2020-09-08T16:48:29.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:48:29.652 INFO:tasks.workunit.client.0.smithi120.stdout:add osd5 3ae8baa2-f695-4ec7-942c-9953e01e86a1 2020-09-08T16:48:29.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDdtVdfG55vJxAA9/I/iJZwA/hqvNmAxWC9hg== 2020-09-08T16:48:29.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDdtVdfG55vJxAA9/I/iJZwA/hqvNmAxWC9hg=="}' 2020-09-08T16:48:29.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3ae8baa2-f695-4ec7-942c-9953e01e86a1 -i td/test-erasure-code/5/new.json 2020-09-08T16:48:30.322 INFO:tasks.workunit.client.0.smithi120.stdout:5 2020-09-08T16:48:30.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/5/new.json 2020-09-08T16:48:30.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdtVdfG55vJxAA9/I/iJZwA/hqvNmAxWC9hg== --osd-uuid 3ae8baa2-f695-4ec7-942c-9953e01e86a1 2020-09-08T16:48:30.353 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:30.358+0000 7f21169e5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:30.354 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:30.359+0000 7f21169e5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:30.356 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:30.361+0000 7f21169e5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:30.357 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:30.362+0000 7f21169e5ec0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2020-09-08T16:48:30.357 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:30.362+0000 7f21169e5ec0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2020-09-08T16:48:30.357 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:30.362+0000 7f21169e5ec0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2020-09-08T16:48:30.425 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:30.427+0000 7f21169e5ec0 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2020-09-08T16:48:35.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/5/keyring 2020-09-08T16:48:35.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:35.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-09-08T16:48:35.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:35.292 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd5 key to auth repository 2020-09-08T16:48:35.791 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.5 2020-09-08T16:48:35.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-09-08T16:48:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:48:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:48:35.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:48:35.811 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:35.815+0000 7f8415edbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:35.813 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:35.817+0000 7f8415edbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:35.814 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:35.819+0000 7f8415edbec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:36.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-09-08T16:48:36.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:48:36.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-09-08T16:48:36.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:48:36.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:48:36.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:36.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:48:36.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:36.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:48:36.175 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:48:36.330 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:36.334+0000 7f8415edbec0 -1 Falling back to public interface 2020-09-08T16:48:36.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:48:37.159 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:37.164+0000 7f8415edbec0 -1 osd.5 0 log_to_monitors {default=true} 2020-09-08T16:48:37.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:48:37.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:37.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:48:37.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:37.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:48:37.540 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:48:38.002 INFO:tasks.workunit.client.0.smithi120.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/41129,v1:127.0.0.1:6843/41129] [v2:127.0.0.1:6844/41129,v1:127.0.0.1:6845/41129] exists,up 3ae8baa2-f695-4ec7-942c-9953e01e86a1 2020-09-08T16:48:38.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:48:38.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:48:38.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:48:38.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:48:38.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2020-09-08T16:48:38.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:48:38.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:48:38.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2020-09-08T16:48:38.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:48:38.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/6 2020-09-08T16:48:38.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:48:38.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:48:38.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:48:38.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:48:38.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2020-09-08T16:48:38.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2020-09-08T16:48:38.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:48:38.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:48:38.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:48:38.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:48:38.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:48:38.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:48:38.008 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:48:38.008 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:48:38.008 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:48:38.008 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:38.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:48:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:48:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:48:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:48:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:48:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:48:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:48:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:48:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:48:38.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/6 2020-09-08T16:48:38.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:48:38.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e0644dc9-0785-4d2c-af34-dd60b583c92c 2020-09-08T16:48:38.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 e0644dc9-0785-4d2c-af34-dd60b583c92c' 2020-09-08T16:48:38.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:48:38.012 INFO:tasks.workunit.client.0.smithi120.stdout:add osd6 e0644dc9-0785-4d2c-af34-dd60b583c92c 2020-09-08T16:48:38.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDmtVdf58lhARAA6WkNvW0W3i/+7OX9HGrJXA== 2020-09-08T16:48:38.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDmtVdf58lhARAA6WkNvW0W3i/+7OX9HGrJXA=="}' 2020-09-08T16:48:38.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e0644dc9-0785-4d2c-af34-dd60b583c92c -i td/test-erasure-code/6/new.json 2020-09-08T16:48:38.541 INFO:tasks.workunit.client.0.smithi120.stdout:6 2020-09-08T16:48:38.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/6/new.json 2020-09-08T16:48:38.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDmtVdf58lhARAA6WkNvW0W3i/+7OX9HGrJXA== --osd-uuid e0644dc9-0785-4d2c-af34-dd60b583c92c 2020-09-08T16:48:38.575 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:38.579+0000 7f2f2d5aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:38.576 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:38.581+0000 7f2f2d5aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:38.582 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:38.583+0000 7f2f2d5aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:38.583 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:38.583+0000 7f2f2d5aeec0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2020-09-08T16:48:38.583 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:38.583+0000 7f2f2d5aeec0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2020-09-08T16:48:38.583 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:38.583+0000 7f2f2d5aeec0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2020-09-08T16:48:38.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:38.623+0000 7f2f2d5aeec0 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2020-09-08T16:48:43.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/6/keyring 2020-09-08T16:48:43.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:43.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2020-09-08T16:48:43.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:43.540 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd6 key to auth repository 2020-09-08T16:48:44.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2020-09-08T16:48:44.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:44.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:48:44.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:48:44.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:48:44.037 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.6 2020-09-08T16:48:44.056 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:44.061+0000 7f25b6dd1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:44.058 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:44.063+0000 7f25b6dd1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:44.059 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:44.064+0000 7f25b6dd1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:44.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2020-09-08T16:48:44.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:48:44.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-09-08T16:48:44.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:48:44.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:48:44.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:44.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:48:44.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:44.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:48:44.406 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:48:44.576 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:44.581+0000 7f25b6dd1ec0 -1 Falling back to public interface 2020-09-08T16:48:44.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:48:45.397 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:45.402+0000 7f25b6dd1ec0 -1 osd.6 0 log_to_monitors {default=true} 2020-09-08T16:48:45.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:48:45.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:45.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:48:45.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:45.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:48:45.775 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:48:46.159 INFO:tasks.workunit.client.0.smithi120.stdout:osd.6 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/42123,v1:127.0.0.1:6851/42123] [v2:127.0.0.1:6852/42123,v1:127.0.0.1:6853/42123] exists,up e0644dc9-0785-4d2c-af34-dd60b583c92c 2020-09-08T16:48:46.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:48:46.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:48:46.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:48:46.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:48:46.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2020-09-08T16:48:46.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:48:46.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:48:46.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=7 2020-09-08T16:48:46.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:48:46.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/7 2020-09-08T16:48:46.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:48:46.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:48:46.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:48:46.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:48:46.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2020-09-08T16:48:46.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2020-09-08T16:48:46.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:48:46.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:48:46.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:48:46.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:48:46.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:48:46.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:48:46.164 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:48:46.164 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:48:46.164 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:48:46.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:46.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:46.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:48:46.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:48:46.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:48:46.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:48:46.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:48:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:48:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:48:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:48:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:48:46.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/7 2020-09-08T16:48:46.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:48:46.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e74f14f4-9594-440c-8f62-216c2e8de7a4 2020-09-08T16:48:46.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd7 e74f14f4-9594-440c-8f62-216c2e8de7a4' 2020-09-08T16:48:46.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:48:46.168 INFO:tasks.workunit.client.0.smithi120.stdout:add osd7 e74f14f4-9594-440c-8f62-216c2e8de7a4 2020-09-08T16:48:46.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDutVdfvdiuChAAcjkeZXjh6XfS3tAqdPrMsg== 2020-09-08T16:48:46.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDutVdfvdiuChAAcjkeZXjh6XfS3tAqdPrMsg=="}' 2020-09-08T16:48:46.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e74f14f4-9594-440c-8f62-216c2e8de7a4 -i td/test-erasure-code/7/new.json 2020-09-08T16:48:46.666 INFO:tasks.workunit.client.0.smithi120.stdout:7 2020-09-08T16:48:46.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/7/new.json 2020-09-08T16:48:46.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 7 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDutVdfvdiuChAAcjkeZXjh6XfS3tAqdPrMsg== --osd-uuid e74f14f4-9594-440c-8f62-216c2e8de7a4 2020-09-08T16:48:46.698 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:46.703+0000 7fb032eb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:46.699 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:46.705+0000 7fb032eb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:46.701 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:46.706+0000 7fb032eb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:46.701 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:46.707+0000 7fb032eb7ec0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2020-09-08T16:48:46.702 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:46.707+0000 7fb032eb7ec0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2020-09-08T16:48:46.702 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:46.707+0000 7fb032eb7ec0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2020-09-08T16:48:46.767 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:46.772+0000 7fb032eb7ec0 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2020-09-08T16:48:51.477 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/7/keyring 2020-09-08T16:48:51.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:51.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd7 key to auth repository 2020-09-08T16:48:51.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:51.479 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd7 key to auth repository 2020-09-08T16:48:51.978 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.7 2020-09-08T16:48:51.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.7 2020-09-08T16:48:51.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 7 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:51.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:48:51.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:48:51.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:48:51.998 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:52.003+0000 7f5983c98ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:51.999 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:52.005+0000 7f5983c98ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:52.000 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:52.006+0000 7f5983c98ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 7 2020-09-08T16:48:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:48:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=7 2020-09-08T16:48:52.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:48:52.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:48:52.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:52.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:48:52.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:52.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2020-09-08T16:48:52.351 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:48:52.517 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:52.522+0000 7f5983c98ec0 -1 Falling back to public interface 2020-09-08T16:48:52.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:48:53.370 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:53.375+0000 7f5983c98ec0 -1 osd.7 0 log_to_monitors {default=true} 2020-09-08T16:48:53.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:48:53.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:48:53.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:48:53.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2020-09-08T16:48:53.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:48:53.722 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:48:54.122 INFO:tasks.workunit.client.0.smithi120.stdout:osd.7 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/43117,v1:127.0.0.1:6859/43117] [v2:127.0.0.1:6860/43117,v1:127.0.0.1:6861/43117] exists,up e74f14f4-9594-440c-8f62-216c2e8de7a4 2020-09-08T16:48:54.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:48:54.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:48:54.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:48:54.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:48:54.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2020-09-08T16:48:54.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:48:54.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:48:54.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=8 2020-09-08T16:48:54.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:48:54.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/8 2020-09-08T16:48:54.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:48:54.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:48:54.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:48:54.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:48:54.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2020-09-08T16:48:54.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2020-09-08T16:48:54.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:48:54.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:48:54.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:48:54.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:48:54.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:48:54.128 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:48:54.128 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:48:54.128 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:48:54.128 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:48:54.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:54.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:48:54.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:48:54.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:48:54.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:48:54.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:48:54.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:48:54.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:48:54.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:48:54.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:48:54.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:48:54.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:48:54.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/8 2020-09-08T16:48:54.132 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:48:54.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0ebc925a-8155-448f-8281-eb81193b4c12 2020-09-08T16:48:54.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd8 0ebc925a-8155-448f-8281-eb81193b4c12' 2020-09-08T16:48:54.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:48:54.133 INFO:tasks.workunit.client.0.smithi120.stdout:add osd8 0ebc925a-8155-448f-8281-eb81193b4c12 2020-09-08T16:48:54.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD2tVdfBMCECBAANjU8vFf4+HqLsFRw8q91Yg== 2020-09-08T16:48:54.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD2tVdfBMCECBAANjU8vFf4+HqLsFRw8q91Yg=="}' 2020-09-08T16:48:54.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0ebc925a-8155-448f-8281-eb81193b4c12 -i td/test-erasure-code/8/new.json 2020-09-08T16:48:54.702 INFO:tasks.workunit.client.0.smithi120.stdout:8 2020-09-08T16:48:54.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/8/new.json 2020-09-08T16:48:54.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 8 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD2tVdfBMCECBAANjU8vFf4+HqLsFRw8q91Yg== --osd-uuid 0ebc925a-8155-448f-8281-eb81193b4c12 2020-09-08T16:48:54.738 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:54.740+0000 7fb42bccdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:54.754 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:54.741+0000 7fb42bccdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:54.754 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:54.743+0000 7fb42bccdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:48:54.755 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:54.744+0000 7fb42bccdec0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2020-09-08T16:48:54.755 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:54.744+0000 7fb42bccdec0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2020-09-08T16:48:54.755 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:54.744+0000 7fb42bccdec0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2020-09-08T16:48:54.814 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:48:54.819+0000 7fb42bccdec0 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2020-09-08T16:48:59.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/8/keyring 2020-09-08T16:48:59.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:48:59.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd8 key to auth repository 2020-09-08T16:48:59.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:48:59.677 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd8 key to auth repository 2020-09-08T16:49:00.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.8 2020-09-08T16:49:00.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:49:00.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 8 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:49:00.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:49:00.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:49:00.187 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.8 2020-09-08T16:49:00.205 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:00.210+0000 7f0780b8cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:00.207 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:00.212+0000 7f0780b8cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:00.208 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:00.213+0000 7f0780b8cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:00.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 8 2020-09-08T16:49:00.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:49:00.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=8 2020-09-08T16:49:00.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:49:00.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:49:00.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:00.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:49:00.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:00.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2020-09-08T16:49:00.565 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:49:00.726 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:00.731+0000 7f0780b8cec0 -1 Falling back to public interface 2020-09-08T16:49:00.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:49:01.504 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:01.509+0000 7f0780b8cec0 -1 osd.8 0 log_to_monitors {default=true} 2020-09-08T16:49:01.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:49:01.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:01.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:49:01.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:01.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2020-09-08T16:49:01.924 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:49:02.365 INFO:tasks.workunit.client.0.smithi120.stdout:osd.8 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/44115,v1:127.0.0.1:6867/44115] [v2:127.0.0.1:6868/44115,v1:127.0.0.1:6869/44115] exists,up 0ebc925a-8155-448f-8281-eb81193b4c12 2020-09-08T16:49:02.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:49:02.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:49:02.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:49:02.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:49:02.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2020-09-08T16:49:02.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:49:02.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:49:02.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=9 2020-09-08T16:49:02.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:49:02.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/9 2020-09-08T16:49:02.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:49:02.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:49:02.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:49:02.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:49:02.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2020-09-08T16:49:02.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2020-09-08T16:49:02.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:49:02.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:49:02.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:49:02.371 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:49:02.372 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:49:02.372 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:49:02.372 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:49:02.372 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:49:02.372 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:49:02.373 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:49:02.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:49:02.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:49:02.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:49:02.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:49:02.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:49:02.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:49:02.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:49:02.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:49:02.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:49:02.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:49:02.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:49:02.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/9 2020-09-08T16:49:02.378 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:49:02.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=52e2c553-eca1-4eca-9ebf-1f8a8c8944f9 2020-09-08T16:49:02.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd9 52e2c553-eca1-4eca-9ebf-1f8a8c8944f9' 2020-09-08T16:49:02.378 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:49:02.379 INFO:tasks.workunit.client.0.smithi120.stdout:add osd9 52e2c553-eca1-4eca-9ebf-1f8a8c8944f9 2020-09-08T16:49:02.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD+tVdfrGwDFxAAG2nlDTbSbQGSE1CvnhZetg== 2020-09-08T16:49:02.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD+tVdfrGwDFxAAG2nlDTbSbQGSE1CvnhZetg=="}' 2020-09-08T16:49:02.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 52e2c553-eca1-4eca-9ebf-1f8a8c8944f9 -i td/test-erasure-code/9/new.json 2020-09-08T16:49:03.006 INFO:tasks.workunit.client.0.smithi120.stdout:9 2020-09-08T16:49:03.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/9/new.json 2020-09-08T16:49:03.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 9 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+tVdfrGwDFxAAG2nlDTbSbQGSE1CvnhZetg== --osd-uuid 52e2c553-eca1-4eca-9ebf-1f8a8c8944f9 2020-09-08T16:49:03.039 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:03.043+0000 7f309afbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:03.041 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:03.045+0000 7f309afbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:03.042 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:03.047+0000 7f309afbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:03.043 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:03.047+0000 7f309afbfec0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2020-09-08T16:49:03.043 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:03.047+0000 7f309afbfec0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2020-09-08T16:49:03.043 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:03.047+0000 7f309afbfec0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2020-09-08T16:49:03.094 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:03.085+0000 7f309afbfec0 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2020-09-08T16:49:08.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/9/keyring 2020-09-08T16:49:08.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:49:08.411 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd9 key to auth repository 2020-09-08T16:49:08.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd9 key to auth repository 2020-09-08T16:49:08.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:49:08.958 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.9 2020-09-08T16:49:08.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.9 2020-09-08T16:49:08.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 9 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:49:08.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:49:08.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:49:08.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:49:08.978 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:08.982+0000 7faa066f8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:08.979 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:08.984+0000 7faa066f8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:08.981 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:08.985+0000 7faa066f8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:09.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 9 2020-09-08T16:49:09.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:49:09.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=9 2020-09-08T16:49:09.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:49:09.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:49:09.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:09.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:49:09.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:09.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2020-09-08T16:49:09.328 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:49:09.500 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:09.504+0000 7faa066f8ec0 -1 Falling back to public interface 2020-09-08T16:49:09.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:49:10.311 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:10.316+0000 7faa066f8ec0 -1 osd.9 0 log_to_monitors {default=true} 2020-09-08T16:49:10.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:49:10.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:10.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:49:10.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:10.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2020-09-08T16:49:10.697 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:49:11.090 INFO:tasks.workunit.client.0.smithi120.stdout:osd.9 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/45109,v1:127.0.0.1:6875/45109] [v2:127.0.0.1:6876/45109,v1:127.0.0.1:6877/45109] exists,up 52e2c553-eca1-4eca-9ebf-1f8a8c8944f9 2020-09-08T16:49:11.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:49:11.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:49:11.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:49:11.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2020-09-08T16:49:11.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2020-09-08T16:49:11.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-code 2020-09-08T16:49:11.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:49:11.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=10 2020-09-08T16:49:11.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:49:11.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-code/10 2020-09-08T16:49:11.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:49:11.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:49:11.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:49:11.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:49:11.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2020-09-08T16:49:11.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2020-09-08T16:49:11.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:49:11.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:49:11.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:49:11.094 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:49:11.094 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:49:11.094 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:49:11.095 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:49:11.095 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:49:11.095 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:49:11.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:49:11.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:49:11.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:49:11.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:49:11.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:49:11.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:49:11.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:49:11.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:49:11.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:49:11.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:49:11.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:49:11.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:49:11.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-code/10 2020-09-08T16:49:11.098 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:49:11.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=735147bd-7bc7-4065-8fd6-f492ad53c4ac 2020-09-08T16:49:11.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd10 735147bd-7bc7-4065-8fd6-f492ad53c4ac' 2020-09-08T16:49:11.100 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:49:11.100 INFO:tasks.workunit.client.0.smithi120.stdout:add osd10 735147bd-7bc7-4065-8fd6-f492ad53c4ac 2020-09-08T16:49:11.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAHtldfOm6OBhAAiPKZ+qH05nC+CDl0HgxpyQ== 2020-09-08T16:49:11.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAHtldfOm6OBhAAiPKZ+qH05nC+CDl0HgxpyQ=="}' 2020-09-08T16:49:11.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 735147bd-7bc7-4065-8fd6-f492ad53c4ac -i td/test-erasure-code/10/new.json 2020-09-08T16:49:11.698 INFO:tasks.workunit.client.0.smithi120.stdout:10 2020-09-08T16:49:11.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-code/10/new.json 2020-09-08T16:49:11.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 10 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAHtldfOm6OBhAAiPKZ+qH05nC+CDl0HgxpyQ== --osd-uuid 735147bd-7bc7-4065-8fd6-f492ad53c4ac 2020-09-08T16:49:11.730 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:11.734+0000 7f5f54fe7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:11.732 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:11.736+0000 7f5f54fe7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:11.733 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:11.737+0000 7f5f54fe7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:11.734 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:11.739+0000 7f5f54fe7ec0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2020-09-08T16:49:11.734 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:11.739+0000 7f5f54fe7ec0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2020-09-08T16:49:11.735 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:11.739+0000 7f5f54fe7ec0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2020-09-08T16:49:11.780 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:11.784+0000 7f5f54fe7ec0 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2020-09-08T16:49:17.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-code/10/keyring 2020-09-08T16:49:17.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:49:17.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd10 key to auth repository 2020-09-08T16:49:17.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:49:17.387 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd10 key to auth repository 2020-09-08T16:49:17.882 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.10 2020-09-08T16:49:17.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.10 2020-09-08T16:49:17.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 10 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:49:17.884 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:49:17.884 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:49:17.885 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:49:17.903 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:17.906+0000 7efce8b0bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:17.903 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:17.908+0000 7efce8b0bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:17.904 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:17.908+0000 7efce8b0bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:18.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 10 2020-09-08T16:49:18.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:49:18.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=10 2020-09-08T16:49:18.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:49:18.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:49:18.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:18.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:49:18.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:18.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2020-09-08T16:49:18.311 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:49:18.421 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:18.425+0000 7efce8b0bec0 -1 Falling back to public interface 2020-09-08T16:49:18.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:49:19.273 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:19.277+0000 7efce8b0bec0 -1 osd.10 0 log_to_monitors {default=true} 2020-09-08T16:49:19.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:49:19.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:19.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:49:19.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:19.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2020-09-08T16:49:19.678 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:49:20.174 INFO:tasks.workunit.client.0.smithi120.stdout:osd.10 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/46103,v1:127.0.0.1:6883/46103] [v2:127.0.0.1:6884/46103,v1:127.0.0.1:6885/46103] exists,up 735147bd-7bc7-4065-8fd6-f492ad53c4ac 2020-09-08T16:49:20.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:49:20.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:49:20.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:49:20.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2020-09-08T16:49:20.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2020-09-08T16:49:20.675 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' does not exist 2020-09-08T16:49:20.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2020-09-08T16:49:20.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T16:49:21.254 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T16:49:21.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:49:22.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2020-09-08T16:49:24.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2020-09-08T16:49:24.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:49:24.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:49:24.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:49:24.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:49:24.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:49:24.724 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:49:24.724 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:49:24.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:49:24.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:49:24.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:49:24.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:49:24.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:49:24.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:49:24.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:49:24.824 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:49:25.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:49:25.207 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:49:25.207 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:49:25.208 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:49:25.208 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:49:25.208 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:49:25.208 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:49:25.209 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:49:25.209 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:49:25.209 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:49:25.210 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:49:25.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:49:25.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:25.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:49:25.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836500 2020-09-08T16:49:25.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836500 2020-09-08T16:49:25.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500' 2020-09-08T16:49:25.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:25.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:49:25.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672978 2020-09-08T16:49:25.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672978 2020-09-08T16:49:25.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978' 2020-09-08T16:49:25.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:25.452 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:49:25.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542161 2020-09-08T16:49:25.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542161 2020-09-08T16:49:25.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161' 2020-09-08T16:49:25.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:25.575 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:49:25.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378639 2020-09-08T16:49:25.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378639 2020-09-08T16:49:25.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639' 2020-09-08T16:49:25.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:25.697 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:49:25.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247822 2020-09-08T16:49:25.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247822 2020-09-08T16:49:25.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639 4-98784247822' 2020-09-08T16:49:25.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:25.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:49:25.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084300 2020-09-08T16:49:25.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084300 2020-09-08T16:49:25.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639 4-98784247822 5-120259084300' 2020-09-08T16:49:25.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:25.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:49:26.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920778 2020-09-08T16:49:26.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920778 2020-09-08T16:49:26.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639 4-98784247822 5-120259084300 6-141733920778' 2020-09-08T16:49:26.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:26.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:49:26.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789961 2020-09-08T16:49:26.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789961 2020-09-08T16:49:26.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639 4-98784247822 5-120259084300 6-141733920778 7-158913789961' 2020-09-08T16:49:26.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:26.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:49:26.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659143 2020-09-08T16:49:26.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659143 2020-09-08T16:49:26.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639 4-98784247822 5-120259084300 6-141733920778 7-158913789961 8-176093659143' 2020-09-08T16:49:26.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:26.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:49:26.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495621 2020-09-08T16:49:26.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495621 2020-09-08T16:49:26.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639 4-98784247822 5-120259084300 6-141733920778 7-158913789961 8-176093659143 9-197568495621' 2020-09-08T16:49:26.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:26.429 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:49:26.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332100 2020-09-08T16:49:26.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332100 2020-09-08T16:49:26.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-60129542161 3-81604378639 4-98784247822 5-120259084300 6-141733920778 7-158913789961 8-176093659143 9-197568495621 10-219043332100' 2020-09-08T16:49:26.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:26.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2020-09-08T16:49:26.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:26.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:49:26.579 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2020-09-08T16:49:26.580 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:26.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836500 2020-09-08T16:49:26.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2020-09-08T16:49:26.580 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 21474836500 2020-09-08T16:49:26.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:49:26.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836500 2020-09-08T16:49:26.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:26.945 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672978 2020-09-08T16:49:26.945 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:26.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:49:26.946 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2020-09-08T16:49:26.947 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:26.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672978 2020-09-08T16:49:26.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-09-08T16:49:26.948 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 42949672978 2020-09-08T16:49:26.949 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:49:27.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672978 2020-09-08T16:49:27.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:27.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542161 2020-09-08T16:49:27.314 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:27.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:49:27.315 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542161 2020-09-08T16:49:27.315 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:27.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542161 2020-09-08T16:49:27.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542161' 2020-09-08T16:49:27.316 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 60129542161 2020-09-08T16:49:27.317 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:49:27.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542161 -lt 60129542161 2020-09-08T16:49:27.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:27.681 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378639 2020-09-08T16:49:27.681 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:27.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:49:27.683 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378639 2020-09-08T16:49:27.683 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:27.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378639 2020-09-08T16:49:27.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378639' 2020-09-08T16:49:27.684 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 81604378639 2020-09-08T16:49:27.685 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:49:28.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378640 -lt 81604378639 2020-09-08T16:49:28.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:28.054 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247822 2020-09-08T16:49:28.054 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:28.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:49:28.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247822 2020-09-08T16:49:28.056 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:28.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247822 2020-09-08T16:49:28.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247822' 2020-09-08T16:49:28.057 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 98784247822 2020-09-08T16:49:28.057 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:49:28.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247822 -lt 98784247822 2020-09-08T16:49:28.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:28.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084300 2020-09-08T16:49:28.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:28.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:49:28.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084300 2020-09-08T16:49:28.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:28.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084300 2020-09-08T16:49:28.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084300' 2020-09-08T16:49:28.428 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 120259084300 2020-09-08T16:49:28.428 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:49:28.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084300 -lt 120259084300 2020-09-08T16:49:28.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:28.798 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-141733920778 2020-09-08T16:49:28.798 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:28.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:49:28.800 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-141733920778 2020-09-08T16:49:28.800 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:28.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920778 2020-09-08T16:49:28.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 141733920778' 2020-09-08T16:49:28.801 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 141733920778 2020-09-08T16:49:28.802 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:49:29.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920779 -lt 141733920778 2020-09-08T16:49:29.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:29.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913789961 2020-09-08T16:49:29.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:29.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:49:29.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913789961 2020-09-08T16:49:29.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:29.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789961 2020-09-08T16:49:29.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913789961' 2020-09-08T16:49:29.171 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.7 seq 158913789961 2020-09-08T16:49:29.171 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:49:29.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789961 -lt 158913789961 2020-09-08T16:49:29.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:29.563 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659143 2020-09-08T16:49:29.564 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:29.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:49:29.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659143 2020-09-08T16:49:29.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:29.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659143 2020-09-08T16:49:29.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659143' 2020-09-08T16:49:29.568 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:49:29.568 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.8 seq 176093659143 2020-09-08T16:49:29.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659144 -lt 176093659143 2020-09-08T16:49:29.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:29.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-197568495621 2020-09-08T16:49:29.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:29.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:49:29.936 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-197568495621 2020-09-08T16:49:29.937 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:29.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495621 2020-09-08T16:49:29.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 197568495621' 2020-09-08T16:49:29.938 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.9 seq 197568495621 2020-09-08T16:49:29.938 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:49:30.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495622 -lt 197568495621 2020-09-08T16:49:30.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:30.322 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332100 2020-09-08T16:49:30.322 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:30.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:49:30.323 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332100 2020-09-08T16:49:30.324 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:30.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332100 2020-09-08T16:49:30.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332100' 2020-09-08T16:49:30.325 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.10 seq 219043332100 2020-09-08T16:49:30.325 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:49:30.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332100 -lt 219043332100 2020-09-08T16:49:30.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:49:30.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:49:30.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:49:31.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 4 == 0 2020-09-08T16:49:31.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:49:31.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:49:31.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:49:31.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:49:31.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:49:31.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:49:31.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:49:31.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:49:31.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:49:31.592 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:49:31.592 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:49:32.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 4 2020-09-08T16:49:32.079 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:49:32.079 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:49:32.080 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2020-09-08T16:49:32.081 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:49:32.081 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:49:32.081 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:49:32.081 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:49:32.081 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:49:32.082 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35290/ceph-osd.0.asok 2020-09-08T16:49:32.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2020-09-08T16:49:32.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.35290/ceph-osd.0.asok log flush 2020-09-08T16:49:32.169 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:49:32.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2020-09-08T16:49:32.178 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:47:56.557+0000 7f99463adec0 0 load: jerasure load: lrc load: isa 2020-09-08T16:49:32.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2020-09-08T16:49:32.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2020-09-08T16:49:32.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2020-09-08T16:49:32.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2020-09-08T16:49:32.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2020-09-08T16:49:33.613 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'ecpool' already exists 2020-09-08T16:49:33.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:49:34.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2020-09-08T16:49:34.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:49:34.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:49:34.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:49:34.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:49:34.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:49:34.624 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:49:34.625 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:49:34.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:49:34.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:49:34.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:49:34.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:49:34.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:49:34.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:49:34.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:49:34.723 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:49:35.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:49:35.339 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:49:35.339 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:49:35.339 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:49:35.339 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:49:35.340 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:49:35.340 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:49:35.340 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:49:35.340 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:49:35.340 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:49:35.341 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:49:35.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:49:35.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:35.342 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:49:35.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836504 2020-09-08T16:49:35.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836504 2020-09-08T16:49:35.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504' 2020-09-08T16:49:35.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:35.465 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:49:35.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672982 2020-09-08T16:49:35.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672982 2020-09-08T16:49:35.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982' 2020-09-08T16:49:35.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:35.588 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:49:35.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542165 2020-09-08T16:49:35.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542165 2020-09-08T16:49:35.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165' 2020-09-08T16:49:35.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:35.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:49:35.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378643 2020-09-08T16:49:35.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378643 2020-09-08T16:49:35.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643' 2020-09-08T16:49:35.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:35.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:49:35.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247826 2020-09-08T16:49:35.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247826 2020-09-08T16:49:35.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643 4-98784247826' 2020-09-08T16:49:35.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:35.999 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:49:36.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084304 2020-09-08T16:49:36.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084304 2020-09-08T16:49:36.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643 4-98784247826 5-120259084304' 2020-09-08T16:49:36.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:36.126 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:49:36.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920782 2020-09-08T16:49:36.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920782 2020-09-08T16:49:36.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643 4-98784247826 5-120259084304 6-141733920782' 2020-09-08T16:49:36.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:36.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:49:36.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789965 2020-09-08T16:49:36.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789965 2020-09-08T16:49:36.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643 4-98784247826 5-120259084304 6-141733920782 7-158913789965' 2020-09-08T16:49:36.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:36.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:49:36.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659147 2020-09-08T16:49:36.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659147 2020-09-08T16:49:36.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643 4-98784247826 5-120259084304 6-141733920782 7-158913789965 8-176093659147' 2020-09-08T16:49:36.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:36.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:49:36.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495626 2020-09-08T16:49:36.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495626 2020-09-08T16:49:36.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643 4-98784247826 5-120259084304 6-141733920782 7-158913789965 8-176093659147 9-197568495626' 2020-09-08T16:49:36.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:36.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:49:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332104 2020-09-08T16:49:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332104 2020-09-08T16:49:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-60129542165 3-81604378643 4-98784247826 5-120259084304 6-141733920782 7-158913789965 8-176093659147 9-197568495626 10-219043332104' 2020-09-08T16:49:36.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:36.876 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836504 2020-09-08T16:49:36.876 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:36.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:49:36.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836504 2020-09-08T16:49:36.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:36.885 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836504 2020-09-08T16:49:36.886 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2020-09-08T16:49:36.886 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:49:36.886 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 21474836504 2020-09-08T16:49:37.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836504 -lt 21474836504 2020-09-08T16:49:37.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:37.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672982 2020-09-08T16:49:37.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:37.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:49:37.256 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672982 2020-09-08T16:49:37.256 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:37.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672982 2020-09-08T16:49:37.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2020-09-08T16:49:37.258 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 42949672982 2020-09-08T16:49:37.258 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:49:37.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672982 -lt 42949672982 2020-09-08T16:49:37.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:37.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542165 2020-09-08T16:49:37.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:37.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:49:37.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542165 2020-09-08T16:49:37.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:37.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542165 2020-09-08T16:49:37.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2020-09-08T16:49:37.632 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 60129542165 2020-09-08T16:49:37.633 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:49:38.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542165 -lt 60129542165 2020-09-08T16:49:38.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:38.050 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378643 2020-09-08T16:49:38.050 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:38.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:49:38.051 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378643 2020-09-08T16:49:38.052 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:38.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378643 2020-09-08T16:49:38.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378643' 2020-09-08T16:49:38.053 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 81604378643 2020-09-08T16:49:38.054 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:49:38.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378644 -lt 81604378643 2020-09-08T16:49:38.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:38.442 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247826 2020-09-08T16:49:38.442 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:38.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:49:38.443 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247826 2020-09-08T16:49:38.443 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:38.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247826 2020-09-08T16:49:38.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247826' 2020-09-08T16:49:38.445 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 98784247826 2020-09-08T16:49:38.445 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:49:38.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247826 -lt 98784247826 2020-09-08T16:49:38.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:38.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084304 2020-09-08T16:49:38.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:38.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:49:38.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084304 2020-09-08T16:49:38.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:38.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084304 2020-09-08T16:49:38.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084304' 2020-09-08T16:49:38.821 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 120259084304 2020-09-08T16:49:38.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:49:39.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084304 -lt 120259084304 2020-09-08T16:49:39.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:39.201 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-141733920782 2020-09-08T16:49:39.202 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:39.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:49:39.203 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-141733920782 2020-09-08T16:49:39.203 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:39.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920782 2020-09-08T16:49:39.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 141733920782' 2020-09-08T16:49:39.204 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 141733920782 2020-09-08T16:49:39.205 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:49:39.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920783 -lt 141733920782 2020-09-08T16:49:39.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:39.589 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913789965 2020-09-08T16:49:39.590 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:39.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:49:39.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913789965 2020-09-08T16:49:39.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:39.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789965 2020-09-08T16:49:39.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913789965' 2020-09-08T16:49:39.593 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.7 seq 158913789965 2020-09-08T16:49:39.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:49:39.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789965 -lt 158913789965 2020-09-08T16:49:39.984 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:39.984 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659147 2020-09-08T16:49:39.985 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:39.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:49:39.986 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659147 2020-09-08T16:49:39.986 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:39.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659147 2020-09-08T16:49:39.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659147' 2020-09-08T16:49:39.988 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:49:39.989 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.8 seq 176093659147 2020-09-08T16:49:40.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659148 -lt 176093659147 2020-09-08T16:49:40.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:40.378 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-197568495626 2020-09-08T16:49:40.379 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:40.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:49:40.380 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-197568495626 2020-09-08T16:49:40.380 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:40.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495626 2020-09-08T16:49:40.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 197568495626' 2020-09-08T16:49:40.381 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.9 seq 197568495626 2020-09-08T16:49:40.382 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:49:40.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495626 -lt 197568495626 2020-09-08T16:49:40.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:40.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332104 2020-09-08T16:49:40.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:40.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:49:40.762 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332104 2020-09-08T16:49:40.763 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:40.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332104 2020-09-08T16:49:40.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332104' 2020-09-08T16:49:40.763 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:49:40.763 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.10 seq 219043332104 2020-09-08T16:49:41.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332104 -lt 219043332104 2020-09-08T16:49:41.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:49:41.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:49:41.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:49:41.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2020-09-08T16:49:41.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:49:41.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:49:41.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:49:41.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:49:41.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:49:41.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:49:41.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:49:42.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2020-09-08T16:49:42.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:49:42.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:49:42.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:49:42.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2020-09-08T16:49:42.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:49:42.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:49:42.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2020-09-08T16:49:42.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-09-08T16:49:42.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2020-09-08T16:49:42.503 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_chunk_mapping 2020-09-08T16:49:42.503 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_put_get_isa 2020-09-08T16:49:42.503 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_put_get_jerasure 2020-09-08T16:49:42.503 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_put_get_lrc_advanced 2020-09-08T16:49:42.503 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_put_get_lrc_kml 2020-09-08T16:49:42.504 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_put_get_shec' 2020-09-08T16:49:42.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2020-09-08T16:49:42.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2020-09-08T16:49:42.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_alignment_constraints: local payload=ABC 2020-09-08T16:49:42.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:250: TEST_alignment_constraints: echo ABC 2020-09-08T16:49:42.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2020-09-08T16:49:42.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: TEST_alignment_constraints: local stripe_unit=4096 2020-09-08T16:49:42.516 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2020-09-08T16:49:42.516 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: grep k= 2020-09-08T16:49:42.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: eval local k=2 2020-09-08T16:49:43.278 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: local k=2 2020-09-08T16:49:43.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:258: TEST_alignment_constraints: local block_size=8191 2020-09-08T16:49:43.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2020-09-08T16:49:43.280 INFO:tasks.workunit.client.0.smithi120.stderr:2+0 records in 2020-09-08T16:49:43.280 INFO:tasks.workunit.client.0.smithi120.stderr:2+0 records out 2020-09-08T16:49:43.280 INFO:tasks.workunit.client.0.smithi120.stderr:16382 bytes (16 kB, 16 KiB) copied, 8.3034e-05 s, 197 MB/s 2020-09-08T16:49:43.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2020-09-08T16:49:43.281 INFO:tasks.workunit.client.0.smithi120.stderr:INFO: op_size has been rounded to 8192 2020-09-08T16:49:43.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:49:43.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2020-09-08T16:49:43.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2020-09-08T16:49:43.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: TEST_chunk_mapping: local dir=td/test-erasure-code 2020-09-08T16:49:43.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2020-09-08T16:49:43.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: verify_chunk_mapping: local dir=td/test-erasure-code 2020-09-08T16:49:43.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:278: verify_chunk_mapping: local poolname=ecpool 2020-09-08T16:49:43.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: verify_chunk_mapping: local first=0 2020-09-08T16:49:43.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:280: verify_chunk_mapping: local second=1 2020-09-08T16:49:43.353 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2020-09-08T16:49:43.354 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2020-09-08T16:49:43.362 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2020-09-08T16:49:43.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2020-09-08T16:49:43.363 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2020-09-08T16:49:43.363 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2020-09-08T16:49:43.375 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2020-09-08T16:49:43.376 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2020-09-08T16:49:43.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2020-09-08T16:49:43.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2020-09-08T16:49:43.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:285: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2020-09-08T16:49:43.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2020-09-08T16:49:43.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2020-09-08T16:49:43.608 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2020-09-08T16:49:43.608 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=ecpool 2020-09-08T16:49:43.609 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHINGecpool 2020-09-08T16:49:43.609 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2020-09-08T16:49:43.609 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:49:44.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:49:44.017 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:49:44.018 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:49:44.018 INFO:tasks.workunit.client.0.smithi120.stderr:0' 2020-09-08T16:49:44.018 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 9 0 2020-09-08T16:49:44.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: local -a osds 2020-09-08T16:49:44.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i = 0 )) 2020-09-08T16:49:44.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2020-09-08T16:49:44.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2020-09-08T16:49:44.101 INFO:tasks.workunit.client.0.smithi120.stderr:admin_socket: exception getting command descriptions: [Errno 13] Permission denied 2020-09-08T16:49:44.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2020-09-08T16:49:44.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2020-09-08T16:49:44.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2020-09-08T16:49:44.185 INFO:tasks.workunit.client.0.smithi120.stderr:admin_socket: exception getting command descriptions: [Errno 13] Permission denied 2020-09-08T16:49:44.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2020-09-08T16:49:44.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2020-09-08T16:49:44.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2020-09-08T16:49:44.269 INFO:tasks.workunit.client.0.smithi120.stderr:admin_socket: exception getting command descriptions: [Errno 13] Permission denied 2020-09-08T16:49:44.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2020-09-08T16:49:44.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2020-09-08T16:49:44.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2020-09-08T16:49:44.353 INFO:tasks.workunit.client.0.smithi120.stderr:admin_socket: exception getting command descriptions: [Errno 13] Permission denied 2020-09-08T16:49:44.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2020-09-08T16:49:44.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2020-09-08T16:49:44.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:49:44.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: rm td/test-erasure-code/COPY 2020-09-08T16:49:44.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2020-09-08T16:49:44.358 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2020-09-08T16:49:44.358 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=ecpool 2020-09-08T16:49:44.359 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHINGecpool 2020-09-08T16:49:44.359 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2020-09-08T16:49:44.359 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:49:44.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:49:44.730 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:49:44.731 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:49:44.731 INFO:tasks.workunit.client.0.smithi120.stderr:0' 2020-09-08T16:49:44.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 9 0 2020-09-08T16:49:44.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: local -a osds 2020-09-08T16:49:44.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2020-09-08T16:49:44.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-code 2020-09-08T16:49:44.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T16:49:44.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=3 2020-09-08T16:49:44.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T16:49:44.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: grep --quiet FIRSTecpool 2020-09-08T16:49:44.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2020-09-08T16:49:44.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-code 2020-09-08T16:49:44.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T16:49:44.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=3 2020-09-08T16:49:44.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T16:49:44.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2020-09-08T16:49:44.734 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:49:44.734 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:49:44.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:49:44.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:49:44.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:49:44.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:49:44.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2020-09-08T16:49:44.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-code 2020-09-08T16:49:44.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T16:49:44.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=3 2020-09-08T16:49:44.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T16:49:44.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2020-09-08T16:49:44.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2020-09-08T16:49:47.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2020-09-08T16:49:47.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-code 2020-09-08T16:49:47.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T16:49:47.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-09-08T16:49:47.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T16:49:47.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-code/3 2020-09-08T16:49:47.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:49:47.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:49:47.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:49:47.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:49:47.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2020-09-08T16:49:47.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2020-09-08T16:49:47.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T16:49:47.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T16:49:47.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:49:47.199 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T16:49:47.200 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:49:47.200 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:49:47.200 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:49:47.200 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:49:47.201 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:49:47.201 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:49:47.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:49:47.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:49:47.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:49:47.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:49:47.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:49:47.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:49:47.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:49:47.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T16:49:47.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T16:49:47.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-code/3 2020-09-08T16:49:47.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-09-08T16:49:47.211 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.3 2020-09-08T16:49:47.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:49:47.212 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-code/3/whoami 2020-09-08T16:49:47.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-09-08T16:49:47.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T16:49:47.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T16:49:47.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T16:49:47.226 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:47.211+0000 7f22e9f5cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:47.228 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:47.213+0000 7f22e9f5cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:47.228 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:47.216+0000 7f22e9f5cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:49:47.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-09-08T16:49:47.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:49:47.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T16:49:47.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:49:47.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:49:47.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:47.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:49:47.568 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:49:47.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:47.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:49:47.726 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:47.731+0000 7f22e9f5cec0 -1 Falling back to public interface 2020-09-08T16:49:47.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:49:48.641 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:49:48.645+0000 7f22e9f5cec0 -1 osd.3 69 log_to_monitors {default=true} 2020-09-08T16:49:48.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:49:48.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:49:48.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:49:48.948 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:49:48.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:49:48.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:49:49.351 INFO:tasks.workunit.client.0.smithi120.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 70 last_clean_interval [19,69) [v2:127.0.0.1:6826/48691,v1:127.0.0.1:6827/48691] [v2:127.0.0.1:6828/48691,v1:127.0.0.1:6829/48691] exists,up 5485caeb-d972-465c-a83e-9237ab4141b7 2020-09-08T16:49:49.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:49:49.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:49:49.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:49:49.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T16:49:49.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:49:49.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:49:49.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:49:49.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:49:49.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:49:49.353 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:49:49.353 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:49:49.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:49:49.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:49:49.354 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:49:49.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:49:49.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:49:49.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:49:49.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:49:49.441 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:49:49.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:49:49.892 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:49:49.892 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:49:49.892 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:49:49.892 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:49:49.892 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:49:49.893 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:49:49.893 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:49:49.893 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:49:49.893 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:49:49.893 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:49:49.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:49:49.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:49.894 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:49:50.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836509 2020-09-08T16:49:50.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836509 2020-09-08T16:49:50.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509' 2020-09-08T16:49:50.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.048 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:49:50.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672987 2020-09-08T16:49:50.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672987 2020-09-08T16:49:50.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987' 2020-09-08T16:49:50.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:49:50.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542170 2020-09-08T16:49:50.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542170 2020-09-08T16:49:50.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170' 2020-09-08T16:49:50.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.358 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:49:50.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=309237645315 2020-09-08T16:49:50.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 309237645315 2020-09-08T16:49:50.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315' 2020-09-08T16:49:50.482 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.483 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:49:50.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247831 2020-09-08T16:49:50.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247831 2020-09-08T16:49:50.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315 4-98784247831' 2020-09-08T16:49:50.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.604 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:49:50.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084309 2020-09-08T16:49:50.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084309 2020-09-08T16:49:50.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315 4-98784247831 5-120259084309' 2020-09-08T16:49:50.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:49:50.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920787 2020-09-08T16:49:50.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920787 2020-09-08T16:49:50.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315 4-98784247831 5-120259084309 6-141733920787' 2020-09-08T16:49:50.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.848 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:49:50.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789970 2020-09-08T16:49:50.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789970 2020-09-08T16:49:50.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315 4-98784247831 5-120259084309 6-141733920787 7-158913789970' 2020-09-08T16:49:50.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:50.971 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:49:51.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659152 2020-09-08T16:49:51.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659152 2020-09-08T16:49:51.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315 4-98784247831 5-120259084309 6-141733920787 7-158913789970 8-176093659152' 2020-09-08T16:49:51.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:51.091 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:49:51.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495630 2020-09-08T16:49:51.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495630 2020-09-08T16:49:51.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315 4-98784247831 5-120259084309 6-141733920787 7-158913789970 8-176093659152 9-197568495630' 2020-09-08T16:49:51.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:49:51.212 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:49:51.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332109 2020-09-08T16:49:51.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332109 2020-09-08T16:49:51.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836509 1-42949672987 2-60129542170 3-309237645315 4-98784247831 5-120259084309 6-141733920787 7-158913789970 8-176093659152 9-197568495630 10-219043332109' 2020-09-08T16:49:51.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:51.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836509 2020-09-08T16:49:51.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:51.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:49:51.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836509 2020-09-08T16:49:51.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:51.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836509 2020-09-08T16:49:51.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2020-09-08T16:49:51.336 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 21474836509 2020-09-08T16:49:51.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:49:51.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836507 -lt 21474836509 2020-09-08T16:49:51.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:49:52.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T16:49:52.714 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:49:53.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836509 -lt 21474836509 2020-09-08T16:49:53.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:53.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672987 2020-09-08T16:49:53.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:53.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:49:53.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672987 2020-09-08T16:49:53.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:53.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672987 2020-09-08T16:49:53.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2020-09-08T16:49:53.090 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 42949672987 2020-09-08T16:49:53.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:49:53.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672988 -lt 42949672987 2020-09-08T16:49:53.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:53.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542170 2020-09-08T16:49:53.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:53.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:49:53.459 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542170 2020-09-08T16:49:53.459 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:53.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542170 2020-09-08T16:49:53.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542170' 2020-09-08T16:49:53.460 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 60129542170 2020-09-08T16:49:53.460 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:49:53.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542170 -lt 60129542170 2020-09-08T16:49:53.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:53.835 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-309237645315 2020-09-08T16:49:53.836 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:53.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:49:53.836 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-309237645315 2020-09-08T16:49:53.836 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:53.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=309237645315 2020-09-08T16:49:53.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 309237645315' 2020-09-08T16:49:53.837 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 309237645315 2020-09-08T16:49:53.837 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:49:54.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 309237645316 -lt 309237645315 2020-09-08T16:49:54.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:54.203 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247831 2020-09-08T16:49:54.203 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:54.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:49:54.204 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247831 2020-09-08T16:49:54.204 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:54.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247831 2020-09-08T16:49:54.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247831' 2020-09-08T16:49:54.205 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.4 seq 98784247831 2020-09-08T16:49:54.206 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:49:54.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247831 -lt 98784247831 2020-09-08T16:49:54.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:54.574 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084309 2020-09-08T16:49:54.575 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:54.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:49:54.575 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084309 2020-09-08T16:49:54.576 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:54.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084309 2020-09-08T16:49:54.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084309' 2020-09-08T16:49:54.577 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.5 seq 120259084309 2020-09-08T16:49:54.577 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:49:54.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084310 -lt 120259084309 2020-09-08T16:49:54.963 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:54.963 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-141733920787 2020-09-08T16:49:54.964 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:54.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:49:54.964 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-141733920787 2020-09-08T16:49:54.965 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:54.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920787 2020-09-08T16:49:54.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 141733920787' 2020-09-08T16:49:54.966 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.6 seq 141733920787 2020-09-08T16:49:54.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:49:55.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920788 -lt 141733920787 2020-09-08T16:49:55.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:55.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913789970 2020-09-08T16:49:55.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:55.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:49:55.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913789970 2020-09-08T16:49:55.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:55.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789970 2020-09-08T16:49:55.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913789970' 2020-09-08T16:49:55.335 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.7 seq 158913789970 2020-09-08T16:49:55.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:49:55.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789970 -lt 158913789970 2020-09-08T16:49:55.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:55.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659152 2020-09-08T16:49:55.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:55.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:49:55.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659152 2020-09-08T16:49:55.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:55.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659152 2020-09-08T16:49:55.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659152' 2020-09-08T16:49:55.732 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.8 seq 176093659152 2020-09-08T16:49:55.732 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:49:56.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659153 -lt 176093659152 2020-09-08T16:49:56.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:56.120 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-197568495630 2020-09-08T16:49:56.121 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:56.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:49:56.122 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-197568495630 2020-09-08T16:49:56.122 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:56.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495630 2020-09-08T16:49:56.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 197568495630' 2020-09-08T16:49:56.123 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.9 seq 197568495630 2020-09-08T16:49:56.123 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:49:56.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495631 -lt 197568495630 2020-09-08T16:49:56.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:49:56.493 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332109 2020-09-08T16:49:56.494 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:49:56.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:49:56.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332109 2020-09-08T16:49:56.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:49:56.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332109 2020-09-08T16:49:56.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332109' 2020-09-08T16:49:56.496 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.10 seq 219043332109 2020-09-08T16:49:56.496 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:49:56.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332110 -lt 219043332109 2020-09-08T16:49:56.894 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:49:56.894 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:49:56.895 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:49:57.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2020-09-08T16:49:57.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:49:57.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:49:57.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:49:57.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:49:57.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:49:57.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:49:57.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:49:57.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2020-09-08T16:49:57.855 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:49:57.856 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:49:57.856 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:49:58.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2020-09-08T16:49:58.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:49:58.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:49:58.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: grep --quiet SECONDecpool 2020-09-08T16:49:58.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2020-09-08T16:49:58.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-code 2020-09-08T16:49:58.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T16:49:58.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=5 2020-09-08T16:49:58.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T16:49:58.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2020-09-08T16:49:58.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-code 2020-09-08T16:49:58.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T16:49:58.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=5 2020-09-08T16:49:58.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T16:49:58.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2020-09-08T16:49:58.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:49:58.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:49:58.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:49:58.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:49:58.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:49:58.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:49:58.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2020-09-08T16:49:58.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-code 2020-09-08T16:49:58.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T16:49:58.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=5 2020-09-08T16:49:58.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T16:49:58.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2020-09-08T16:49:58.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2020-09-08T16:50:01.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2020-09-08T16:50:01.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-code 2020-09-08T16:50:01.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T16:50:01.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=5 2020-09-08T16:50:01.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T16:50:01.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-code/5 2020-09-08T16:50:01.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:50:01.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:50:01.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:50:01.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:50:01.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2020-09-08T16:50:01.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2020-09-08T16:50:01.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T16:50:01.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T16:50:01.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:50:01.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T16:50:01.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:50:01.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:50:01.362 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:50:01.363 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:50:01.363 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:50:01.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:50:01.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:50:01.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:50:01.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:50:01.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:50:01.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:50:01.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:50:01.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:50:01.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T16:50:01.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T16:50:01.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-code/5 2020-09-08T16:50:01.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.5 2020-09-08T16:50:01.365 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.5 2020-09-08T16:50:01.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 5 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:50:01.367 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-code/5/whoami 2020-09-08T16:50:01.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 5 = 5 ']' 2020-09-08T16:50:01.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T16:50:01.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T16:50:01.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T16:50:01.382 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:01.386+0000 7fdcc1881ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:01.384 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:01.388+0000 7fdcc1881ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:01.386 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:01.390+0000 7fdcc1881ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:01.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 5 2020-09-08T16:50:01.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:50:01.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-09-08T16:50:01.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:50:01.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:50:01.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:50:01.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:50:01.745 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:50:01.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:50:01.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:50:01.909 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:01.913+0000 7fdcc1881ec0 -1 Falling back to public interface 2020-09-08T16:50:02.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:50:02.917 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:02.922+0000 7fdcc1881ec0 -1 osd.5 74 log_to_monitors {default=true} 2020-09-08T16:50:03.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:50:03.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:50:03.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:50:03.135 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:03.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:50:03.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:50:03.698 INFO:tasks.workunit.client.0.smithi120.stderr:osd.5 up in weight 1 up_from 78 up_thru 78 down_at 75 last_clean_interval [28,74) [v2:127.0.0.1:6842/50194,v1:127.0.0.1:6843/50194] [v2:127.0.0.1:6844/50194,v1:127.0.0.1:6845/50194] exists,up 3ae8baa2-f695-4ec7-942c-9953e01e86a1 2020-09-08T16:50:03.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:50:03.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:50:03.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:50:03.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T16:50:03.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:50:03.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:50:03.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:50:03.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:50:03.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:50:03.701 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:50:03.701 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:50:03.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:50:03.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:50:03.710 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:50:03.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:50:03.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:50:03.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:50:03.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:50:03.787 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:50:04.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:50:04.423 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:04.423 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:50:04.423 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:50:04.424 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:50:04.424 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:50:04.424 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:50:04.424 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:50:04.424 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:50:04.425 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:50:04.425 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:50:04.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:50:04.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:04.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:50:04.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836514 2020-09-08T16:50:04.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836514 2020-09-08T16:50:04.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514' 2020-09-08T16:50:04.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:04.545 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:50:04.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672992 2020-09-08T16:50:04.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672992 2020-09-08T16:50:04.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992' 2020-09-08T16:50:04.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:04.794 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:50:04.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542175 2020-09-08T16:50:04.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542175 2020-09-08T16:50:04.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175' 2020-09-08T16:50:04.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:04.917 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:50:05.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=309237645320 2020-09-08T16:50:05.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 309237645320 2020-09-08T16:50:05.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320' 2020-09-08T16:50:05.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:05.046 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:50:05.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247836 2020-09-08T16:50:05.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247836 2020-09-08T16:50:05.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320 4-98784247836' 2020-09-08T16:50:05.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:05.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:50:05.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449091 2020-09-08T16:50:05.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449091 2020-09-08T16:50:05.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320 4-98784247836 5-335007449091' 2020-09-08T16:50:05.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:05.291 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:50:05.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920792 2020-09-08T16:50:05.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920792 2020-09-08T16:50:05.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320 4-98784247836 5-335007449091 6-141733920792' 2020-09-08T16:50:05.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:05.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:50:05.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789975 2020-09-08T16:50:05.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789975 2020-09-08T16:50:05.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320 4-98784247836 5-335007449091 6-141733920792 7-158913789975' 2020-09-08T16:50:05.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:05.552 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:50:05.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659157 2020-09-08T16:50:05.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659157 2020-09-08T16:50:05.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320 4-98784247836 5-335007449091 6-141733920792 7-158913789975 8-176093659157' 2020-09-08T16:50:05.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:05.676 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:50:05.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495635 2020-09-08T16:50:05.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495635 2020-09-08T16:50:05.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320 4-98784247836 5-335007449091 6-141733920792 7-158913789975 8-176093659157 9-197568495635' 2020-09-08T16:50:05.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:05.797 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:50:05.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332114 2020-09-08T16:50:05.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332114 2020-09-08T16:50:05.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836514 1-42949672992 2-60129542175 3-309237645320 4-98784247836 5-335007449091 6-141733920792 7-158913789975 8-176093659157 9-197568495635 10-219043332114' 2020-09-08T16:50:05.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:05.927 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836514 2020-09-08T16:50:05.927 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:05.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:50:05.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836514 2020-09-08T16:50:05.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:05.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836514 2020-09-08T16:50:05.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2020-09-08T16:50:05.930 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 21474836514 2020-09-08T16:50:05.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:50:06.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836514 -lt 21474836514 2020-09-08T16:50:06.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:06.282 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672992 2020-09-08T16:50:06.283 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:06.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:50:06.283 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672992 2020-09-08T16:50:06.284 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:06.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672992 2020-09-08T16:50:06.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2020-09-08T16:50:06.285 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 42949672992 2020-09-08T16:50:06.285 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:50:06.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672992 -lt 42949672992 2020-09-08T16:50:06.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:06.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542175 2020-09-08T16:50:06.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:06.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:50:06.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542175 2020-09-08T16:50:06.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:06.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542175 2020-09-08T16:50:06.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542175' 2020-09-08T16:50:06.657 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 60129542175 2020-09-08T16:50:06.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:50:07.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542175 -lt 60129542175 2020-09-08T16:50:07.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:07.064 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-309237645320 2020-09-08T16:50:07.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:07.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:50:07.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-309237645320 2020-09-08T16:50:07.066 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:07.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=309237645320 2020-09-08T16:50:07.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 309237645320' 2020-09-08T16:50:07.067 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 309237645320 2020-09-08T16:50:07.067 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:50:07.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 309237645320 -lt 309237645320 2020-09-08T16:50:07.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:07.432 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247836 2020-09-08T16:50:07.432 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:07.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:50:07.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247836 2020-09-08T16:50:07.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:07.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247836 2020-09-08T16:50:07.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247836' 2020-09-08T16:50:07.434 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.4 seq 98784247836 2020-09-08T16:50:07.434 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:50:07.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247836 -lt 98784247836 2020-09-08T16:50:07.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:07.805 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-335007449091 2020-09-08T16:50:07.805 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:07.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:50:07.806 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-335007449091 2020-09-08T16:50:07.806 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:07.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449091 2020-09-08T16:50:07.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 335007449091' 2020-09-08T16:50:07.807 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.5 seq 335007449091 2020-09-08T16:50:07.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:50:08.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449091 -lt 335007449091 2020-09-08T16:50:08.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:08.198 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-141733920792 2020-09-08T16:50:08.198 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:08.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:50:08.200 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-141733920792 2020-09-08T16:50:08.200 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:08.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920792 2020-09-08T16:50:08.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 141733920792' 2020-09-08T16:50:08.201 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.6 seq 141733920792 2020-09-08T16:50:08.201 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:50:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920793 -lt 141733920792 2020-09-08T16:50:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:08.569 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913789975 2020-09-08T16:50:08.569 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:08.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:50:08.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913789975 2020-09-08T16:50:08.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789975 2020-09-08T16:50:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913789975' 2020-09-08T16:50:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.7 seq 158913789975 2020-09-08T16:50:08.572 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:50:08.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789975 -lt 158913789975 2020-09-08T16:50:08.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:08.942 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659157 2020-09-08T16:50:08.942 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:08.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:50:08.943 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659157 2020-09-08T16:50:08.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:08.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659157 2020-09-08T16:50:08.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659157' 2020-09-08T16:50:08.944 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.8 seq 176093659157 2020-09-08T16:50:08.945 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:50:09.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659158 -lt 176093659157 2020-09-08T16:50:09.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:09.311 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-197568495635 2020-09-08T16:50:09.311 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:09.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:50:09.312 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-197568495635 2020-09-08T16:50:09.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:09.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495635 2020-09-08T16:50:09.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 197568495635' 2020-09-08T16:50:09.314 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.9 seq 197568495635 2020-09-08T16:50:09.314 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:50:09.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495636 -lt 197568495635 2020-09-08T16:50:09.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:09.711 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332114 2020-09-08T16:50:09.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:09.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:50:09.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332114 2020-09-08T16:50:09.713 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:09.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332114 2020-09-08T16:50:09.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332114' 2020-09-08T16:50:09.714 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.10 seq 219043332114 2020-09-08T16:50:09.714 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:50:10.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332114 -lt 219043332114 2020-09-08T16:50:10.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:50:10.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:50:10.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:50:10.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2020-09-08T16:50:10.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:50:10.627 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:50:10.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:50:10.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:50:10.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:50:10.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:50:10.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:50:10.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2020-09-08T16:50:10.980 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:50:10.980 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:50:10.980 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:50:11.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2020-09-08T16:50:11.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:50:11.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:50:11.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2020-09-08T16:50:11.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:314: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2020-09-08T16:50:12.372 INFO:tasks.workunit.client.0.smithi120.stdout:crush-device-class= 2020-09-08T16:50:12.372 INFO:tasks.workunit.client.0.smithi120.stdout:crush-failure-domain=host 2020-09-08T16:50:12.373 INFO:tasks.workunit.client.0.smithi120.stdout:crush-root=default 2020-09-08T16:50:12.373 INFO:tasks.workunit.client.0.smithi120.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2020-09-08T16:50:12.373 INFO:tasks.workunit.client.0.smithi120.stdout:k=-1 2020-09-08T16:50:12.373 INFO:tasks.workunit.client.0.smithi120.stdout:l=-1 2020-09-08T16:50:12.373 INFO:tasks.workunit.client.0.smithi120.stdout:layers=[ [ "cDD", "" ] ] 2020-09-08T16:50:12.374 INFO:tasks.workunit.client.0.smithi120.stdout:m=-1 2020-09-08T16:50:12.374 INFO:tasks.workunit.client.0.smithi120.stdout:mapping=_DD 2020-09-08T16:50:12.374 INFO:tasks.workunit.client.0.smithi120.stdout:plugin=lrc 2020-09-08T16:50:12.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:315: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2020-09-08T16:50:12.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2020-09-08T16:50:13.228 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'remap-pool' already exists 2020-09-08T16:50:13.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:50:14.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:323: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2020-09-08T16:50:14.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: verify_chunk_mapping: local dir=td/test-erasure-code 2020-09-08T16:50:14.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:278: verify_chunk_mapping: local poolname=remap-pool 2020-09-08T16:50:14.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: verify_chunk_mapping: local first=1 2020-09-08T16:50:14.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:280: verify_chunk_mapping: local second=2 2020-09-08T16:50:14.239 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2020-09-08T16:50:14.239 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2020-09-08T16:50:14.252 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2020-09-08T16:50:14.252 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2020-09-08T16:50:14.253 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2020-09-08T16:50:14.253 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2020-09-08T16:50:14.265 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2020-09-08T16:50:14.341 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2020-09-08T16:50:14.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2020-09-08T16:50:14.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2020-09-08T16:50:14.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:285: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2020-09-08T16:50:15.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2020-09-08T16:50:16.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2020-09-08T16:50:16.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2020-09-08T16:50:16.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=remap-pool 2020-09-08T16:50:16.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHINGremap-pool 2020-09-08T16:50:16.328 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2020-09-08T16:50:16.328 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:50:16.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=4 2020-09-08T16:50:16.721 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:50:16.721 INFO:tasks.workunit.client.0.smithi120.stderr:9' 2020-09-08T16:50:16.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 4 6 9 2020-09-08T16:50:16.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: local -a osds 2020-09-08T16:50:16.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i = 0 )) 2020-09-08T16:50:16.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2020-09-08T16:50:16.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2020-09-08T16:50:16.806 INFO:tasks.workunit.client.0.smithi120.stderr:admin_socket: exception getting command descriptions: [Errno 13] Permission denied 2020-09-08T16:50:16.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2020-09-08T16:50:16.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2020-09-08T16:50:16.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2020-09-08T16:50:16.891 INFO:tasks.workunit.client.0.smithi120.stderr:admin_socket: exception getting command descriptions: [Errno 13] Permission denied 2020-09-08T16:50:16.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2020-09-08T16:50:16.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2020-09-08T16:50:16.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2020-09-08T16:50:16.977 INFO:tasks.workunit.client.0.smithi120.stderr:admin_socket: exception getting command descriptions: [Errno 13] Permission denied 2020-09-08T16:50:16.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2020-09-08T16:50:16.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2020-09-08T16:50:16.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:50:16.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: rm td/test-erasure-code/COPY 2020-09-08T16:50:16.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2020-09-08T16:50:16.980 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2020-09-08T16:50:16.980 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=remap-pool 2020-09-08T16:50:16.980 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHINGremap-pool 2020-09-08T16:50:16.981 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2020-09-08T16:50:16.981 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:50:17.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=4 2020-09-08T16:50:17.368 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:50:17.368 INFO:tasks.workunit.client.0.smithi120.stderr:9' 2020-09-08T16:50:17.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 4 6 9 2020-09-08T16:50:17.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: local -a osds 2020-09-08T16:50:17.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:17.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-code 2020-09-08T16:50:17.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T16:50:17.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=6 2020-09-08T16:50:17.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T16:50:17.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2020-09-08T16:50:17.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:17.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-code 2020-09-08T16:50:17.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T16:50:17.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=6 2020-09-08T16:50:17.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T16:50:17.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2020-09-08T16:50:17.371 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:50:17.372 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:50:17.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:50:17.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:50:17.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:50:17.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:50:17.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:17.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-code 2020-09-08T16:50:17.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T16:50:17.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=6 2020-09-08T16:50:17.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T16:50:17.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2020-09-08T16:50:17.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:20.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2020-09-08T16:50:20.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-code 2020-09-08T16:50:20.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T16:50:20.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=6 2020-09-08T16:50:20.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T16:50:20.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-code/6 2020-09-08T16:50:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:50:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:50:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:50:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:50:20.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2020-09-08T16:50:20.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2020-09-08T16:50:20.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T16:50:20.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T16:50:20.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:50:20.062 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T16:50:20.063 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:50:20.063 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:50:20.064 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:50:20.064 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:50:20.064 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:50:20.064 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:50:20.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:50:20.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:50:20.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:50:20.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:50:20.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:50:20.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:50:20.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:50:20.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T16:50:20.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T16:50:20.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-code/6 2020-09-08T16:50:20.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.6 2020-09-08T16:50:20.067 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.6 2020-09-08T16:50:20.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 6 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:50:20.067 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-code/6/whoami 2020-09-08T16:50:20.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 6 = 6 ']' 2020-09-08T16:50:20.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T16:50:20.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T16:50:20.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T16:50:20.080 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:20.085+0000 7fc4c31b6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:20.081 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:20.087+0000 7fc4c31b6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:20.083 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:20.089+0000 7fc4c31b6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:20.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 6 2020-09-08T16:50:20.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:50:20.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-09-08T16:50:20.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:50:20.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:50:20.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:50:20.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:50:20.445 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:50:20.446 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:50:20.446 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:50:20.598 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:20.603+0000 7fc4c31b6ec0 -1 Falling back to public interface 2020-09-08T16:50:20.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:50:21.538 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:21.543+0000 7fc4c31b6ec0 -1 osd.6 91 log_to_monitors {default=true} 2020-09-08T16:50:21.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:50:21.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:50:21.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:50:21.850 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:21.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:50:21.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:50:22.348 INFO:tasks.workunit.client.0.smithi120.stderr:osd.6 up in weight 1 up_from 94 up_thru 84 down_at 92 last_clean_interval [33,91) [v2:127.0.0.1:6850/51869,v1:127.0.0.1:6851/51869] [v2:127.0.0.1:6852/51869,v1:127.0.0.1:6853/51869] exists,up e0644dc9-0785-4d2c-af34-dd60b583c92c 2020-09-08T16:50:22.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:50:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:50:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:50:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T16:50:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:50:22.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:50:22.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:50:22.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:50:22.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:50:22.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:50:22.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:50:22.351 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:50:22.351 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:50:22.351 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:50:22.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:50:22.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:50:22.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:50:22.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:50:22.439 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:50:22.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:50:22.860 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:22.861 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:50:22.861 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:50:22.861 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:50:22.861 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:50:22.862 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:50:22.862 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:50:22.862 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:50:22.862 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:50:22.863 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:50:22.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:50:22.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:22.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:50:22.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836519 2020-09-08T16:50:22.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836519 2020-09-08T16:50:22.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519' 2020-09-08T16:50:22.984 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:22.984 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:50:23.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672998 2020-09-08T16:50:23.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672998 2020-09-08T16:50:23.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998' 2020-09-08T16:50:23.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:23.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:50:23.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542180 2020-09-08T16:50:23.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542180 2020-09-08T16:50:23.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180' 2020-09-08T16:50:23.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:23.292 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:50:23.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=309237645325 2020-09-08T16:50:23.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 309237645325 2020-09-08T16:50:23.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325' 2020-09-08T16:50:23.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:23.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:50:23.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247841 2020-09-08T16:50:23.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247841 2020-09-08T16:50:23.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325 4-98784247841' 2020-09-08T16:50:23.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:23.587 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:50:23.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449097 2020-09-08T16:50:23.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449097 2020-09-08T16:50:23.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325 4-98784247841 5-335007449097' 2020-09-08T16:50:23.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:23.710 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:50:23.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=403726925827 2020-09-08T16:50:23.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 403726925827 2020-09-08T16:50:23.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325 4-98784247841 5-335007449097 6-403726925827' 2020-09-08T16:50:23.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:23.833 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:50:23.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789980 2020-09-08T16:50:23.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789980 2020-09-08T16:50:23.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325 4-98784247841 5-335007449097 6-403726925827 7-158913789980' 2020-09-08T16:50:23.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:23.957 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:50:24.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659163 2020-09-08T16:50:24.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659163 2020-09-08T16:50:24.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325 4-98784247841 5-335007449097 6-403726925827 7-158913789980 8-176093659163' 2020-09-08T16:50:24.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:24.138 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:50:24.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495641 2020-09-08T16:50:24.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495641 2020-09-08T16:50:24.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325 4-98784247841 5-335007449097 6-403726925827 7-158913789980 8-176093659163 9-197568495641' 2020-09-08T16:50:24.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:24.300 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:50:24.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332119 2020-09-08T16:50:24.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332119 2020-09-08T16:50:24.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542180 3-309237645325 4-98784247841 5-335007449097 6-403726925827 7-158913789980 8-176093659163 9-197568495641 10-219043332119' 2020-09-08T16:50:24.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:24.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836519 2020-09-08T16:50:24.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:24.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:50:24.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836519 2020-09-08T16:50:24.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:24.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836519 2020-09-08T16:50:24.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2020-09-08T16:50:24.425 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 21474836519 2020-09-08T16:50:24.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:50:24.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836520 -lt 21474836519 2020-09-08T16:50:24.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:24.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672998 2020-09-08T16:50:24.792 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:24.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:50:24.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672998 2020-09-08T16:50:24.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:24.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672998 2020-09-08T16:50:24.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2020-09-08T16:50:24.794 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 42949672998 2020-09-08T16:50:24.794 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:50:25.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672998 -lt 42949672998 2020-09-08T16:50:25.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:25.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542180 2020-09-08T16:50:25.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:25.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:50:25.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542180 2020-09-08T16:50:25.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:25.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542180 2020-09-08T16:50:25.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2020-09-08T16:50:25.171 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 60129542180 2020-09-08T16:50:25.171 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:50:25.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542180 -lt 60129542180 2020-09-08T16:50:25.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:25.536 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-309237645325 2020-09-08T16:50:25.536 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:25.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:50:25.538 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-309237645325 2020-09-08T16:50:25.538 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:25.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=309237645325 2020-09-08T16:50:25.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 309237645325' 2020-09-08T16:50:25.539 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 309237645325 2020-09-08T16:50:25.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:50:25.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 309237645326 -lt 309237645325 2020-09-08T16:50:25.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:25.937 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247841 2020-09-08T16:50:25.938 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:25.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:50:25.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247841 2020-09-08T16:50:25.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:25.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247841 2020-09-08T16:50:25.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247841' 2020-09-08T16:50:25.940 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.4 seq 98784247841 2020-09-08T16:50:25.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:50:26.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247842 -lt 98784247841 2020-09-08T16:50:26.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:26.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-335007449097 2020-09-08T16:50:26.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:26.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:50:26.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-335007449097 2020-09-08T16:50:26.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:26.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449097 2020-09-08T16:50:26.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 335007449097' 2020-09-08T16:50:26.334 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.5 seq 335007449097 2020-09-08T16:50:26.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:50:26.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449097 -lt 335007449097 2020-09-08T16:50:26.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:26.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-403726925827 2020-09-08T16:50:26.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:26.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:50:26.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-403726925827 2020-09-08T16:50:26.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:26.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=403726925827 2020-09-08T16:50:26.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 403726925827' 2020-09-08T16:50:26.727 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.6 seq 403726925827 2020-09-08T16:50:26.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:50:27.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 403726925827 -lt 403726925827 2020-09-08T16:50:27.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:27.098 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913789980 2020-09-08T16:50:27.098 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:27.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:50:27.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913789980 2020-09-08T16:50:27.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:27.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789980 2020-09-08T16:50:27.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913789980' 2020-09-08T16:50:27.101 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.7 seq 158913789980 2020-09-08T16:50:27.101 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:50:27.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789981 -lt 158913789980 2020-09-08T16:50:27.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:27.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659163 2020-09-08T16:50:27.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:27.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:50:27.475 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659163 2020-09-08T16:50:27.475 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:27.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659163 2020-09-08T16:50:27.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659163' 2020-09-08T16:50:27.476 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.8 seq 176093659163 2020-09-08T16:50:27.476 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:50:27.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659163 -lt 176093659163 2020-09-08T16:50:27.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:27.840 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-197568495641 2020-09-08T16:50:27.840 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:27.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:50:27.841 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-197568495641 2020-09-08T16:50:27.841 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:27.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495641 2020-09-08T16:50:27.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 197568495641' 2020-09-08T16:50:27.843 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.9 seq 197568495641 2020-09-08T16:50:27.843 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:50:28.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495642 -lt 197568495641 2020-09-08T16:50:28.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:28.209 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332119 2020-09-08T16:50:28.209 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:28.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:50:28.210 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332119 2020-09-08T16:50:28.210 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:28.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332119 2020-09-08T16:50:28.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332119' 2020-09-08T16:50:28.211 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.10 seq 219043332119 2020-09-08T16:50:28.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:50:28.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332120 -lt 219043332119 2020-09-08T16:50:28.584 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:50:28.585 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:50:28.585 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:50:29.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 28 == 0 2020-09-08T16:50:29.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:50:29.075 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:50:29.075 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:50:29.076 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:50:29.076 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:50:29.076 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:50:29.076 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:50:29.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=28 2020-09-08T16:50:29.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:50:29.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:50:29.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:50:29.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 28 = 28 2020-09-08T16:50:29.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:50:29.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:50:29.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:29.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-code 2020-09-08T16:50:29.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: grep --quiet SECONDremap-pool 2020-09-08T16:50:29.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T16:50:29.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=9 2020-09-08T16:50:29.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T16:50:29.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:29.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-code 2020-09-08T16:50:29.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T16:50:29.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=9 2020-09-08T16:50:29.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T16:50:29.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2020-09-08T16:50:29.950 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:50:29.951 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:50:29.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:50:29.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:50:29.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:50:30.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:50:30.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:30.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-code 2020-09-08T16:50:30.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T16:50:30.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=9 2020-09-08T16:50:30.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T16:50:30.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2020-09-08T16:50:30.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2020-09-08T16:50:33.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2020-09-08T16:50:33.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-code 2020-09-08T16:50:33.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T16:50:33.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=9 2020-09-08T16:50:33.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T16:50:33.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-code/9 2020-09-08T16:50:33.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2020-09-08T16:50:33.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:50:33.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:50:33.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:50:33.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2020-09-08T16:50:33.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2020-09-08T16:50:33.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T16:50:33.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T16:50:33.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2020-09-08T16:50:33.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T16:50:33.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:50:33.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:50:33.015 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:50:33.015 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:50:33.015 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:50:33.016 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:50:33.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' 2020-09-08T16:50:33.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:50:33.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2020-09-08T16:50:33.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2020-09-08T16:50:33.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:50:33.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:50:33.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:50:33.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T16:50:33.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T16:50:33.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-code/9 2020-09-08T16:50:33.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.9 2020-09-08T16:50:33.021 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.9 2020-09-08T16:50:33.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 9 --fsid=c2c4acc5-f158-4185-8a23-31cffee0e4c9 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.35290/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:50:33.021 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-code/9/whoami 2020-09-08T16:50:33.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 9 = 9 ']' 2020-09-08T16:50:33.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T16:50:33.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T16:50:33.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T16:50:33.030 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:33.034+0000 7f0aef731ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:33.032 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:33.037+0000 7f0aef731ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:33.035 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:33.039+0000 7f0aef731ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:50:33.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 9 2020-09-08T16:50:33.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:50:33.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=9 2020-09-08T16:50:33.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:50:33.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:50:33.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:50:33.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:50:33.390 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:50:33.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:50:33.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2020-09-08T16:50:33.550 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:33.555+0000 7f0aef731ec0 -1 Falling back to public interface 2020-09-08T16:50:33.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:50:34.604 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:50:34.608+0000 7f0aef731ec0 -1 osd.9 96 log_to_monitors {default=true} 2020-09-08T16:50:34.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:50:34.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:50:34.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:50:34.758 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:34.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2020-09-08T16:50:34.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:50:35.223 INFO:tasks.workunit.client.0.smithi120.stderr:osd.9 up in weight 1 up_from 100 up_thru 62 down_at 97 last_clean_interval [46,96) [v2:127.0.0.1:6874/53346,v1:127.0.0.1:6875/53346] [v2:127.0.0.1:6876/53346,v1:127.0.0.1:6877/53346] exists,up 52e2c553-eca1-4eca-9ebf-1f8a8c8944f9 2020-09-08T16:50:35.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:50:35.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:50:35.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:50:35.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T16:50:35.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:50:35.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:50:35.225 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:50:35.225 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:50:35.225 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:50:35.225 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:50:35.225 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:50:35.226 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:50:35.226 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:50:35.226 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:50:35.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:50:35.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:50:35.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:50:35.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:50:35.314 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:50:36.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:50:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:50:36.026 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:50:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:50:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:50:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:50:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:50:36.027 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:50:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:50:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:50:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:50:36.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:36.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:50:36.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836524 2020-09-08T16:50:36.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836524 2020-09-08T16:50:36.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524' 2020-09-08T16:50:36.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:36.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:50:36.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673002 2020-09-08T16:50:36.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673002 2020-09-08T16:50:36.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2020-09-08T16:50:36.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:36.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:50:36.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542185 2020-09-08T16:50:36.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542185 2020-09-08T16:50:36.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185' 2020-09-08T16:50:36.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:36.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:50:36.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=309237645330 2020-09-08T16:50:36.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 309237645330 2020-09-08T16:50:36.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330' 2020-09-08T16:50:36.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:36.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:50:36.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247846 2020-09-08T16:50:36.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247846 2020-09-08T16:50:36.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330 4-98784247846' 2020-09-08T16:50:36.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:36.768 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:50:36.886 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449101 2020-09-08T16:50:36.887 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449101 2020-09-08T16:50:36.887 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330 4-98784247846 5-335007449101' 2020-09-08T16:50:36.887 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:36.888 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:50:37.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=403726925832 2020-09-08T16:50:37.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 403726925832 2020-09-08T16:50:37.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330 4-98784247846 5-335007449101 6-403726925832' 2020-09-08T16:50:37.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:37.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:50:37.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789985 2020-09-08T16:50:37.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789985 2020-09-08T16:50:37.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330 4-98784247846 5-335007449101 6-403726925832 7-158913789985' 2020-09-08T16:50:37.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:37.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:50:37.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659167 2020-09-08T16:50:37.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659167 2020-09-08T16:50:37.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330 4-98784247846 5-335007449101 6-403726925832 7-158913789985 8-176093659167' 2020-09-08T16:50:37.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:37.260 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:50:37.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=429496729603 2020-09-08T16:50:37.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 429496729603 2020-09-08T16:50:37.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330 4-98784247846 5-335007449101 6-403726925832 7-158913789985 8-176093659167 9-429496729603' 2020-09-08T16:50:37.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:50:37.379 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:50:37.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332124 2020-09-08T16:50:37.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332124 2020-09-08T16:50:37.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-60129542185 3-309237645330 4-98784247846 5-335007449101 6-403726925832 7-158913789985 8-176093659167 9-429496729603 10-219043332124' 2020-09-08T16:50:37.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:37.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836524 2020-09-08T16:50:37.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:37.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:50:37.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836524 2020-09-08T16:50:37.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836524 2020-09-08T16:50:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2020-09-08T16:50:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 21474836524 2020-09-08T16:50:37.506 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:50:37.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836522 -lt 21474836524 2020-09-08T16:50:37.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:50:38.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T16:50:38.871 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:50:39.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836524 -lt 21474836524 2020-09-08T16:50:39.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:39.258 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673002 2020-09-08T16:50:39.259 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:39.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:50:39.260 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673002 2020-09-08T16:50:39.260 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:39.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673002 2020-09-08T16:50:39.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2020-09-08T16:50:39.260 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 42949673002 2020-09-08T16:50:39.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:50:39.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673003 -lt 42949673002 2020-09-08T16:50:39.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:39.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542185 2020-09-08T16:50:39.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:39.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:50:39.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542185 2020-09-08T16:50:39.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:39.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542185 2020-09-08T16:50:39.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542185' 2020-09-08T16:50:39.632 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 60129542185 2020-09-08T16:50:39.633 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:50:39.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542185 -lt 60129542185 2020-09-08T16:50:39.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:40.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-309237645330 2020-09-08T16:50:40.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:40.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:50:40.001 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-309237645330 2020-09-08T16:50:40.001 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:40.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=309237645330 2020-09-08T16:50:40.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 309237645330' 2020-09-08T16:50:40.003 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 309237645330 2020-09-08T16:50:40.003 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:50:40.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 309237645331 -lt 309237645330 2020-09-08T16:50:40.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:40.374 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247846 2020-09-08T16:50:40.374 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:40.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:50:40.375 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247846 2020-09-08T16:50:40.375 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:40.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247846 2020-09-08T16:50:40.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247846' 2020-09-08T16:50:40.376 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.4 seq 98784247846 2020-09-08T16:50:40.376 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:50:40.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247846 -lt 98784247846 2020-09-08T16:50:40.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:40.768 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-335007449101 2020-09-08T16:50:40.769 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:40.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:50:40.770 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-335007449101 2020-09-08T16:50:40.770 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:40.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449101 2020-09-08T16:50:40.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 335007449101' 2020-09-08T16:50:40.770 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.5 seq 335007449101 2020-09-08T16:50:40.771 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:50:41.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449102 -lt 335007449101 2020-09-08T16:50:41.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:41.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-403726925832 2020-09-08T16:50:41.137 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:41.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:50:41.138 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-403726925832 2020-09-08T16:50:41.138 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:41.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=403726925832 2020-09-08T16:50:41.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 403726925832' 2020-09-08T16:50:41.139 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.6 seq 403726925832 2020-09-08T16:50:41.139 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:50:41.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 403726925832 -lt 403726925832 2020-09-08T16:50:41.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:41.537 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913789985 2020-09-08T16:50:41.538 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:41.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:50:41.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913789985 2020-09-08T16:50:41.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:41.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789985 2020-09-08T16:50:41.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913789985' 2020-09-08T16:50:41.540 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.7 seq 158913789985 2020-09-08T16:50:41.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:50:41.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789986 -lt 158913789985 2020-09-08T16:50:41.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:41.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659167 2020-09-08T16:50:41.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:41.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:50:41.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659167 2020-09-08T16:50:41.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:41.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659167 2020-09-08T16:50:41.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659167' 2020-09-08T16:50:41.932 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.8 seq 176093659167 2020-09-08T16:50:41.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:50:42.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659168 -lt 176093659167 2020-09-08T16:50:42.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:42.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-429496729603 2020-09-08T16:50:42.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:42.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:50:42.307 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-429496729603 2020-09-08T16:50:42.307 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:42.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=429496729603 2020-09-08T16:50:42.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 429496729603' 2020-09-08T16:50:42.308 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.9 seq 429496729603 2020-09-08T16:50:42.308 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:50:42.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 429496729604 -lt 429496729603 2020-09-08T16:50:42.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:50:42.677 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332124 2020-09-08T16:50:42.677 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:50:42.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:50:42.678 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332124 2020-09-08T16:50:42.678 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:50:42.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332124 2020-09-08T16:50:42.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332124' 2020-09-08T16:50:42.679 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.10 seq 219043332124 2020-09-08T16:50:42.679 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:50:43.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332125 -lt 219043332124 2020-09-08T16:50:43.045 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:50:43.046 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:50:43.046 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:50:43.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 28 == 0 2020-09-08T16:50:43.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:50:43.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:50:43.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:50:43.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:50:43.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:50:43.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:50:43.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:50:43.977 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=28 2020-09-08T16:50:43.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:50:43.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:50:43.978 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:50:44.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 28 = 28 2020-09-08T16:50:44.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:50:44.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:50:44.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:325: TEST_chunk_mapping: delete_pool remap-pool 2020-09-08T16:50:44.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=remap-pool 2020-09-08T16:50:44.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2020-09-08T16:50:44.950 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'remap-pool' does not exist 2020-09-08T16:50:44.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:326: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2020-09-08T16:50:45.531 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile remap-profile does not exist 2020-09-08T16:50:45.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2020-09-08T16:50:45.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2020-09-08T16:50:45.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2020-09-08T16:50:45.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: erasure_code_plugin_exists: local plugin=isa 2020-09-08T16:50:45.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1922: erasure_code_plugin_exists: local status 2020-09-08T16:50:45.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: erasure_code_plugin_exists: local grepstr 2020-09-08T16:50:45.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1924: erasure_code_plugin_exists: local s 2020-09-08T16:50:45.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: erasure_code_plugin_exists: case `uname` in 2020-09-08T16:50:45.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: erasure_code_plugin_exists: uname 2020-09-08T16:50:45.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: erasure_code_plugin_exists: grepstr='isa.*No such file' 2020-09-08T16:50:45.542 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1930: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2020-09-08T16:50:46.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1930: erasure_code_plugin_exists: s= 2020-09-08T16:50:46.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1931: erasure_code_plugin_exists: local status=0 2020-09-08T16:50:46.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2020-09-08T16:50:46.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2020-09-08T16:50:46.859 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile TESTPROFILE does not exist 2020-09-08T16:50:46.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: erasure_code_plugin_exists: return 0 2020-09-08T16:50:46.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2020-09-08T16:50:46.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:194: TEST_rados_put_get_isa: local poolname=pool-isa 2020-09-08T16:50:46.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:196: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2020-09-08T16:50:47.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:199: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2020-09-08T16:50:47.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2020-09-08T16:50:48.237 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-isa' already exists 2020-09-08T16:50:48.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:50:49.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:202: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2020-09-08T16:50:49.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2020-09-08T16:50:49.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2020-09-08T16:50:49.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2020-09-08T16:50:49.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:49.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2020-09-08T16:50:49.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:49.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2020-09-08T16:50:49.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:49.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2020-09-08T16:50:49.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:49.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2020-09-08T16:50:49.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2020-09-08T16:50:50.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:50:50.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:50:50.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2020-09-08T16:50:50.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:50:50.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2020-09-08T16:50:50.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-isa 2020-09-08T16:50:50.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:50:50.699 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-isa SOMETHING 2020-09-08T16:50:50.699 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:50:51.077 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=4 2020-09-08T16:50:51.078 INFO:tasks.workunit.client.0.smithi120.stderr:10 2020-09-08T16:50:51.078 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:50:51.078 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:50:51.078 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:50:51.079 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:50:51.079 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:50:51.079 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:51.080 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:50:51.080 INFO:tasks.workunit.client.0.smithi120.stderr:9' 2020-09-08T16:50:51.080 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2020-09-08T16:50:51.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2020-09-08T16:50:51.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2020-09-08T16:50:51.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2020-09-08T16:50:51.647 INFO:tasks.workunit.client.0.smithi120.stderr:osd.9 is already out. 2020-09-08T16:50:51.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-isa SOMETHING 2020-09-08T16:50:51.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-isa 2020-09-08T16:50:51.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:50:51.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<9\>' 2020-09-08T16:50:51.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-isa SOMETHING 2020-09-08T16:50:51.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:50:52.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=4 2020-09-08T16:50:52.073 INFO:tasks.workunit.client.0.smithi120.stderr:10 2020-09-08T16:50:52.073 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:50:52.073 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:50:52.073 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:50:52.074 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:50:52.074 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:50:52.074 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:50:52.074 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:50:52.074 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T16:50:52.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2020-09-08T16:50:52.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:50:53.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:50:53.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 9 2020-09-08T16:50:54.186 INFO:tasks.workunit.client.0.smithi120.stderr:osd.9 is already in. 2020-09-08T16:50:54.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:50:54.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:204: TEST_rados_put_get_isa: delete_pool pool-isa 2020-09-08T16:50:54.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=pool-isa 2020-09-08T16:50:54.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2020-09-08T16:50:55.492 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-isa' does not exist 2020-09-08T16:50:55.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2020-09-08T16:50:55.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2020-09-08T16:50:55.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2020-09-08T16:50:55.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:210: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2020-09-08T16:50:55.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2020-09-08T16:50:55.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2020-09-08T16:50:55.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2020-09-08T16:50:55.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:55.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2020-09-08T16:50:55.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:55.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2020-09-08T16:50:55.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:55.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2020-09-08T16:50:55.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:50:55.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2020-09-08T16:50:55.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2020-09-08T16:50:56.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:50:56.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:50:56.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2020-09-08T16:50:56.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:50:56.267 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2020-09-08T16:50:56.268 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=ecpool 2020-09-08T16:50:56.268 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:50:56.268 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map ecpool SOMETHING 2020-09-08T16:50:56.269 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:50:56.703 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=6 2020-09-08T16:50:56.703 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:50:56.703 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:50:56.704 INFO:tasks.workunit.client.0.smithi120.stderr:9' 2020-09-08T16:50:56.704 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 6 0 7 9 2020-09-08T16:50:56.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2020-09-08T16:50:56.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2020-09-08T16:50:56.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2020-09-08T16:50:57.316 INFO:tasks.workunit.client.0.smithi120.stderr:osd.9 is already out. 2020-09-08T16:50:57.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds ecpool SOMETHING 2020-09-08T16:50:57.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=ecpool 2020-09-08T16:50:57.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:50:57.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<9\>' 2020-09-08T16:50:57.326 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map ecpool SOMETHING 2020-09-08T16:50:57.326 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:50:57.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=6 2020-09-08T16:50:57.996 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:50:57.996 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:50:57.996 INFO:tasks.workunit.client.0.smithi120.stderr:5' 2020-09-08T16:50:57.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 6 0 7 5 2020-09-08T16:50:57.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:50:59.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:50:59.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 9 2020-09-08T16:50:59.993 INFO:tasks.workunit.client.0.smithi120.stderr:osd.9 is already in. 2020-09-08T16:51:00.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:51:00.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2020-09-08T16:51:00.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:213: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2020-09-08T16:51:00.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:215: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2020-09-08T16:51:01.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2020-09-08T16:51:01.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2020-09-08T16:51:03.148 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T16:51:03.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:51:04.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:222: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2020-09-08T16:51:04.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2020-09-08T16:51:04.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2020-09-08T16:51:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2020-09-08T16:51:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2020-09-08T16:51:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:04.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2020-09-08T16:51:04.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:04.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2020-09-08T16:51:04.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:04.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2020-09-08T16:51:04.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2020-09-08T16:51:06.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:51:06.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:51:06.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2020-09-08T16:51:06.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:51:06.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2020-09-08T16:51:06.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:51:06.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:06.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2020-09-08T16:51:06.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:07.450 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=5 2020-09-08T16:51:07.450 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:51:07.451 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:07.452 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:07.452 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:07.453 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:07.453 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 5 1 9 4 3 10 2020-09-08T16:51:07.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2020-09-08T16:51:07.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2020-09-08T16:51:07.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2020-09-08T16:51:08.731 INFO:tasks.workunit.client.0.smithi120.stderr:osd.10 is already out. 2020-09-08T16:51:08.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-jerasure SOMETHING 2020-09-08T16:51:08.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:51:08.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:08.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<10\>' 2020-09-08T16:51:08.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2020-09-08T16:51:08.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:09.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=5 2020-09-08T16:51:09.179 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:51:09.179 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:09.179 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:09.179 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:09.179 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T16:51:09.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 5 1 9 4 3 6 2020-09-08T16:51:09.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:51:09.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:51:09.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 10 2020-09-08T16:51:10.549 INFO:tasks.workunit.client.0.smithi120.stderr:osd.10 is already in. 2020-09-08T16:51:10.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:51:10.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2020-09-08T16:51:10.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_osds_out_in: local dir=td/test-erasure-code 2020-09-08T16:51:10.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:101: rados_osds_out_in: local poolname=pool-jerasure 2020-09-08T16:51:10.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:102: rados_osds_out_in: local objname=SOMETHING 2020-09-08T16:51:10.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2020-09-08T16:51:10.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 FFFF 2020-09-08T16:51:10.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2020-09-08T16:51:10.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 GGGG 2020-09-08T16:51:10.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2020-09-08T16:51:10.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 HHHH 2020-09-08T16:51:10.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2020-09-08T16:51:10.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 IIII 2020-09-08T16:51:10.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:112: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2020-09-08T16:51:14.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:113: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:51:14.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:114: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:51:14.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:115: rados_osds_out_in: rm td/test-erasure-code/COPY 2020-09-08T16:51:14.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:123: rados_osds_out_in: wait_for_clean 2020-09-08T16:51:14.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:51:14.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:51:14.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:51:14.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:51:14.064 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:51:14.064 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:51:14.064 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:51:14.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:51:14.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:51:14.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:51:14.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:51:14.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:51:14.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:51:14.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:51:14.159 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:51:14.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:51:14.784 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:51:14.784 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:51:14.784 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:14.784 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:14.785 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:51:14.785 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:51:14.785 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:51:14.785 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:51:14.786 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:14.786 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:14.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:51:14.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:14.787 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:51:14.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836534 2020-09-08T16:51:14.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836534 2020-09-08T16:51:14.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534' 2020-09-08T16:51:14.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:14.950 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:51:15.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673012 2020-09-08T16:51:15.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673012 2020-09-08T16:51:15.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2020-09-08T16:51:15.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:15.073 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:51:15.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542195 2020-09-08T16:51:15.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542195 2020-09-08T16:51:15.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195' 2020-09-08T16:51:15.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:15.320 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:51:15.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=309237645340 2020-09-08T16:51:15.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 309237645340 2020-09-08T16:51:15.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340' 2020-09-08T16:51:15.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:15.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:51:15.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247856 2020-09-08T16:51:15.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247856 2020-09-08T16:51:15.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340 4-98784247856' 2020-09-08T16:51:15.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:15.627 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:51:15.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449111 2020-09-08T16:51:15.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449111 2020-09-08T16:51:15.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340 4-98784247856 5-335007449111' 2020-09-08T16:51:15.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:15.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:51:16.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=403726925841 2020-09-08T16:51:16.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 403726925841 2020-09-08T16:51:16.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340 4-98784247856 5-335007449111 6-403726925841' 2020-09-08T16:51:16.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:16.154 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:51:16.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789995 2020-09-08T16:51:16.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789995 2020-09-08T16:51:16.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340 4-98784247856 5-335007449111 6-403726925841 7-158913789995' 2020-09-08T16:51:16.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:16.610 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:51:16.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659177 2020-09-08T16:51:16.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659177 2020-09-08T16:51:16.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340 4-98784247856 5-335007449111 6-403726925841 7-158913789995 8-176093659177' 2020-09-08T16:51:16.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:16.814 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:51:17.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=429496729613 2020-09-08T16:51:17.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 429496729613 2020-09-08T16:51:17.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340 4-98784247856 5-335007449111 6-403726925841 7-158913789995 8-176093659177 9-429496729613' 2020-09-08T16:51:17.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:17.112 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:51:17.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332134 2020-09-08T16:51:17.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332134 2020-09-08T16:51:17.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-60129542195 3-309237645340 4-98784247856 5-335007449111 6-403726925841 7-158913789995 8-176093659177 9-429496729613 10-219043332134' 2020-09-08T16:51:17.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:17.326 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836534 2020-09-08T16:51:17.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:17.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:51:17.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836534 2020-09-08T16:51:17.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:17.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836534 2020-09-08T16:51:17.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2020-09-08T16:51:17.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:51:17.330 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 21474836534 2020-09-08T16:51:17.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836534 -lt 21474836534 2020-09-08T16:51:17.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:17.776 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673012 2020-09-08T16:51:17.777 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:17.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:51:17.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673012 2020-09-08T16:51:17.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673012 2020-09-08T16:51:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2020-09-08T16:51:17.779 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 42949673012 2020-09-08T16:51:17.780 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:51:18.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673012 -lt 42949673012 2020-09-08T16:51:18.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:18.408 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542195 2020-09-08T16:51:18.409 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:18.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:51:18.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542195 2020-09-08T16:51:18.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:18.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542195 2020-09-08T16:51:18.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542195' 2020-09-08T16:51:18.411 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 60129542195 2020-09-08T16:51:18.411 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:51:18.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542195 -lt 60129542195 2020-09-08T16:51:18.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:18.850 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-309237645340 2020-09-08T16:51:18.851 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:18.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:51:18.852 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-309237645340 2020-09-08T16:51:18.852 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:18.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=309237645340 2020-09-08T16:51:18.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 309237645340' 2020-09-08T16:51:18.854 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 309237645340 2020-09-08T16:51:18.854 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:51:19.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 309237645340 -lt 309237645340 2020-09-08T16:51:19.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:19.227 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247856 2020-09-08T16:51:19.227 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:19.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:51:19.228 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247856 2020-09-08T16:51:19.228 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:19.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247856 2020-09-08T16:51:19.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247856' 2020-09-08T16:51:19.229 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:51:19.229 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 98784247856 2020-09-08T16:51:19.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247856 -lt 98784247856 2020-09-08T16:51:19.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:19.615 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-335007449111 2020-09-08T16:51:19.616 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:19.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:51:19.627 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-335007449111 2020-09-08T16:51:19.627 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:19.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449111 2020-09-08T16:51:19.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 335007449111' 2020-09-08T16:51:19.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:51:19.628 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 335007449111 2020-09-08T16:51:20.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449111 -lt 335007449111 2020-09-08T16:51:20.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:20.035 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-403726925841 2020-09-08T16:51:20.036 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:20.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:51:20.037 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-403726925841 2020-09-08T16:51:20.037 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:20.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=403726925841 2020-09-08T16:51:20.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 403726925841' 2020-09-08T16:51:20.039 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:51:20.040 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 403726925841 2020-09-08T16:51:20.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 403726925842 -lt 403726925841 2020-09-08T16:51:20.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:20.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913789995 2020-09-08T16:51:20.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:20.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:51:20.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913789995 2020-09-08T16:51:20.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:20.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789995 2020-09-08T16:51:20.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913789995' 2020-09-08T16:51:20.426 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.7 seq 158913789995 2020-09-08T16:51:20.427 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:51:20.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789996 -lt 158913789995 2020-09-08T16:51:20.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:20.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659177 2020-09-08T16:51:20.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:20.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:51:20.794 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659177 2020-09-08T16:51:20.795 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:20.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659177 2020-09-08T16:51:20.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659177' 2020-09-08T16:51:20.798 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.8 seq 176093659177 2020-09-08T16:51:20.799 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:51:21.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659178 -lt 176093659177 2020-09-08T16:51:21.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:21.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-429496729613 2020-09-08T16:51:21.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:21.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:51:21.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-429496729613 2020-09-08T16:51:21.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:21.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=429496729613 2020-09-08T16:51:21.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 429496729613' 2020-09-08T16:51:21.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:51:21.169 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.9 seq 429496729613 2020-09-08T16:51:21.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 429496729614 -lt 429496729613 2020-09-08T16:51:21.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:21.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332134 2020-09-08T16:51:21.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:21.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:51:21.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332134 2020-09-08T16:51:21.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:21.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332134 2020-09-08T16:51:21.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332134' 2020-09-08T16:51:21.541 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.10 seq 219043332134 2020-09-08T16:51:21.542 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:51:21.910 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332134 -lt 219043332134 2020-09-08T16:51:21.911 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:51:21.911 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:21.911 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 28 == 0 2020-09-08T16:51:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:51:22.398 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:51:22.398 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:51:22.399 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:51:22.399 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:51:22.399 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:51:22.399 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:51:22.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=25 2020-09-08T16:51:22.770 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:51:22.770 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:22.770 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:23.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 25 = 28 2020-09-08T16:51:23.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 25 '!=' -1 2020-09-08T16:51:23.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T16:51:23.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=25 2020-09-08T16:51:23.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:51:23.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:51:23.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:51:23.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:51:23.356 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:51:23.357 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:51:23.357 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:51:23.357 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:51:23.357 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:51:23.358 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:51:23.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=25 2020-09-08T16:51:23.717 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:51:23.717 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:23.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:24.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 25 = 28 2020-09-08T16:51:24.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 25 '!=' 25 2020-09-08T16:51:24.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:51:24.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:51:24.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:51:24.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:51:24.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:51:24.202 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:51:24.202 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:51:24.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-09-08T16:51:24.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-09-08T16:51:24.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2020-09-08T16:51:24.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:51:24.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-09-08T16:51:24.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:51:24.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:51:24.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:51:24.926 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:51:24.926 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:51:24.926 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:51:24.926 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:51:24.927 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:51:25.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=28 2020-09-08T16:51:25.284 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:51:25.284 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:25.284 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:25.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 28 = 28 2020-09-08T16:51:25.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:51:25.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:51:25.792 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:124: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2020-09-08T16:51:25.792 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:51:25.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:25.793 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2020-09-08T16:51:25.793 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:26.177 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=5 2020-09-08T16:51:26.177 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:51:26.178 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:26.178 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:26.178 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:26.178 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:26.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 5 1 9 4 3 10 2020-09-08T16:51:26.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:124: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2020-09-08T16:51:26.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:125: rados_osds_out_in: osds=($osds_list) 2020-09-08T16:51:26.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:125: rados_osds_out_in: local -a osds 2020-09-08T16:51:26.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:126: rados_osds_out_in: for osd in 0 1 2020-09-08T16:51:26.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: ceph osd out 5 2020-09-08T16:51:26.844 INFO:tasks.workunit.client.0.smithi120.stderr:osd.5 is already out. 2020-09-08T16:51:26.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:126: rados_osds_out_in: for osd in 0 1 2020-09-08T16:51:26.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: ceph osd out 1 2020-09-08T16:51:28.460 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 is already out. 2020-09-08T16:51:28.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: wait_for_clean 2020-09-08T16:51:28.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:51:28.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:51:28.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:51:28.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:51:28.469 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:51:28.469 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:51:28.470 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:51:28.470 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:51:28.470 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:51:28.470 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:51:28.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:51:28.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:51:28.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:51:28.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:51:28.558 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:51:29.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:51:29.395 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:51:29.395 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:51:29.395 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:29.395 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:29.395 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:51:29.396 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:51:29.396 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:51:29.396 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:51:29.396 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:29.396 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:29.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:51:29.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:29.397 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:51:29.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836539 2020-09-08T16:51:29.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836539 2020-09-08T16:51:29.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539' 2020-09-08T16:51:29.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:29.638 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:51:29.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673017 2020-09-08T16:51:29.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673017 2020-09-08T16:51:29.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017' 2020-09-08T16:51:29.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:29.958 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:51:30.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542200 2020-09-08T16:51:30.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542200 2020-09-08T16:51:30.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200' 2020-09-08T16:51:30.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:30.205 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:51:30.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=309237645345 2020-09-08T16:51:30.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 309237645345 2020-09-08T16:51:30.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345' 2020-09-08T16:51:30.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:30.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:51:30.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247861 2020-09-08T16:51:30.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247861 2020-09-08T16:51:30.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345 4-98784247861' 2020-09-08T16:51:30.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:30.496 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:51:30.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449116 2020-09-08T16:51:30.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449116 2020-09-08T16:51:30.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345 4-98784247861 5-335007449116' 2020-09-08T16:51:30.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:30.759 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:51:30.885 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=403726925846 2020-09-08T16:51:30.885 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 403726925846 2020-09-08T16:51:30.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345 4-98784247861 5-335007449116 6-403726925846' 2020-09-08T16:51:30.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:30.888 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:51:31.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913790000 2020-09-08T16:51:31.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913790000 2020-09-08T16:51:31.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345 4-98784247861 5-335007449116 6-403726925846 7-158913790000' 2020-09-08T16:51:31.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:31.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:51:31.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659182 2020-09-08T16:51:31.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659182 2020-09-08T16:51:31.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345 4-98784247861 5-335007449116 6-403726925846 7-158913790000 8-176093659182' 2020-09-08T16:51:31.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:31.131 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:51:31.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=429496729618 2020-09-08T16:51:31.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 429496729618 2020-09-08T16:51:31.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345 4-98784247861 5-335007449116 6-403726925846 7-158913790000 8-176093659182 9-429496729618' 2020-09-08T16:51:31.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:31.254 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:51:31.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332139 2020-09-08T16:51:31.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332139 2020-09-08T16:51:31.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836539 1-42949673017 2-60129542200 3-309237645345 4-98784247861 5-335007449116 6-403726925846 7-158913790000 8-176093659182 9-429496729618 10-219043332139' 2020-09-08T16:51:31.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:31.400 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836539 2020-09-08T16:51:31.400 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:31.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:51:31.401 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836539 2020-09-08T16:51:31.401 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:31.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836539 2020-09-08T16:51:31.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2020-09-08T16:51:31.402 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 21474836539 2020-09-08T16:51:31.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:51:32.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836539 -lt 21474836539 2020-09-08T16:51:32.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:32.112 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673017 2020-09-08T16:51:32.113 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:32.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:51:32.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673017 2020-09-08T16:51:32.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:32.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673017 2020-09-08T16:51:32.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2020-09-08T16:51:32.116 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 42949673017 2020-09-08T16:51:32.116 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:51:32.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673017 -lt 42949673017 2020-09-08T16:51:32.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:32.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542200 2020-09-08T16:51:32.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:32.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:51:32.506 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542200 2020-09-08T16:51:32.507 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:32.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542200 2020-09-08T16:51:32.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542200' 2020-09-08T16:51:32.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:51:32.508 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 60129542200 2020-09-08T16:51:32.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542200 -lt 60129542200 2020-09-08T16:51:32.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:32.874 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-309237645345 2020-09-08T16:51:32.874 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:32.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:51:32.876 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-309237645345 2020-09-08T16:51:32.876 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:32.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=309237645345 2020-09-08T16:51:32.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 309237645345' 2020-09-08T16:51:32.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:51:32.878 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 309237645345 2020-09-08T16:51:33.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 309237645345 -lt 309237645345 2020-09-08T16:51:33.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:33.246 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247861 2020-09-08T16:51:33.246 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:33.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:51:33.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247861 2020-09-08T16:51:33.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:33.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247861 2020-09-08T16:51:33.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247861' 2020-09-08T16:51:33.252 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 98784247861 2020-09-08T16:51:33.253 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:51:33.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247861 -lt 98784247861 2020-09-08T16:51:33.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:33.641 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-335007449116 2020-09-08T16:51:33.641 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:33.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:51:33.642 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-335007449116 2020-09-08T16:51:33.642 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:33.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449116 2020-09-08T16:51:33.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 335007449116' 2020-09-08T16:51:33.644 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:51:33.644 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 335007449116 2020-09-08T16:51:34.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449116 -lt 335007449116 2020-09-08T16:51:34.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:34.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-403726925846 2020-09-08T16:51:34.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:34.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:51:34.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-403726925846 2020-09-08T16:51:34.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:34.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=403726925846 2020-09-08T16:51:34.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 403726925846' 2020-09-08T16:51:34.016 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 403726925846 2020-09-08T16:51:34.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:51:34.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 403726925847 -lt 403726925846 2020-09-08T16:51:34.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:34.399 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913790000 2020-09-08T16:51:34.399 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:34.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:51:34.400 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913790000 2020-09-08T16:51:34.401 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:34.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913790000 2020-09-08T16:51:34.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913790000' 2020-09-08T16:51:34.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:51:34.403 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.7 seq 158913790000 2020-09-08T16:51:34.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913790000 -lt 158913790000 2020-09-08T16:51:34.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:34.784 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659182 2020-09-08T16:51:34.785 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:34.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:51:34.785 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659182 2020-09-08T16:51:34.785 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:34.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659182 2020-09-08T16:51:34.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659182' 2020-09-08T16:51:34.786 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:51:34.787 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.8 seq 176093659182 2020-09-08T16:51:35.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659183 -lt 176093659182 2020-09-08T16:51:35.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:35.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-429496729618 2020-09-08T16:51:35.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:35.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:51:35.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-429496729618 2020-09-08T16:51:35.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:35.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=429496729618 2020-09-08T16:51:35.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 429496729618' 2020-09-08T16:51:35.169 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.9 seq 429496729618 2020-09-08T16:51:35.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:51:35.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 429496729618 -lt 429496729618 2020-09-08T16:51:35.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:35.567 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332139 2020-09-08T16:51:35.567 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:35.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:51:35.568 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332139 2020-09-08T16:51:35.568 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:35.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332139 2020-09-08T16:51:35.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332139' 2020-09-08T16:51:35.570 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:51:35.570 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.10 seq 219043332139 2020-09-08T16:51:35.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332139 -lt 219043332139 2020-09-08T16:51:35.940 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:51:35.940 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:35.940 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:36.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 28 == 0 2020-09-08T16:51:36.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:51:36.457 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:51:36.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:51:36.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:51:36.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:51:36.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:51:36.459 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:51:36.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=28 2020-09-08T16:51:36.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:51:36.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:36.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:37.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 28 = 28 2020-09-08T16:51:37.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:51:37.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:51:37.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: for osd in 0 1 2020-09-08T16:51:37.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2020-09-08T16:51:37.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:51:37.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:37.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: grep '\<5\>' 2020-09-08T16:51:37.320 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2020-09-08T16:51:37.320 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:37.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2020-09-08T16:51:37.749 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:51:37.750 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:37.750 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:37.750 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:37.751 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:37.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 7 9 4 3 10 2020-09-08T16:51:37.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: for osd in 0 1 2020-09-08T16:51:37.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2020-09-08T16:51:37.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:51:37.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:37.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: grep '\<1\>' 2020-09-08T16:51:37.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2020-09-08T16:51:37.753 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:38.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2020-09-08T16:51:38.120 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:51:38.120 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:38.120 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:38.120 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:38.120 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:38.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 7 9 4 3 10 2020-09-08T16:51:38.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:136: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:51:38.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:51:38.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:143: rados_osds_out_in: for osd in 0 1 2020-09-08T16:51:38.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:144: rados_osds_out_in: ceph osd in 5 2020-09-08T16:51:38.662 INFO:tasks.workunit.client.0.smithi120.stderr:osd.5 is already in. 2020-09-08T16:51:38.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:143: rados_osds_out_in: for osd in 0 1 2020-09-08T16:51:38.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:144: rados_osds_out_in: ceph osd in 1 2020-09-08T16:51:40.162 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 is already in. 2020-09-08T16:51:40.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:146: rados_osds_out_in: wait_for_clean 2020-09-08T16:51:40.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:51:40.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:51:40.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:51:40.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:51:40.172 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:51:40.172 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:51:40.172 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:51:40.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:51:40.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:51:40.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:51:40.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:51:40.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:51:40.261 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:51:40.261 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:51:40.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:51:40.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:51:40.671 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:51:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:51:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:51:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:51:40.673 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:51:40.673 INFO:tasks.workunit.client.0.smithi120.stderr:8 2020-09-08T16:51:40.673 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:40.673 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:40.673 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:51:40.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:40.674 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:51:40.982 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836543 2020-09-08T16:51:40.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836543 2020-09-08T16:51:40.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543' 2020-09-08T16:51:40.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:40.984 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:51:41.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673021 2020-09-08T16:51:41.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673021 2020-09-08T16:51:41.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021' 2020-09-08T16:51:41.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:41.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:51:41.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542204 2020-09-08T16:51:41.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542204 2020-09-08T16:51:41.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204' 2020-09-08T16:51:41.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:41.230 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:51:41.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=309237645349 2020-09-08T16:51:41.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 309237645349 2020-09-08T16:51:41.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349' 2020-09-08T16:51:41.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:41.546 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:51:41.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247865 2020-09-08T16:51:41.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247865 2020-09-08T16:51:41.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349 4-98784247865' 2020-09-08T16:51:41.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:41.671 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:51:41.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=335007449120 2020-09-08T16:51:41.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 335007449120 2020-09-08T16:51:41.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349 4-98784247865 5-335007449120' 2020-09-08T16:51:41.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:41.810 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:51:41.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=403726925851 2020-09-08T16:51:41.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 403726925851 2020-09-08T16:51:41.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349 4-98784247865 5-335007449120 6-403726925851' 2020-09-08T16:51:41.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:41.936 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:51:42.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913790004 2020-09-08T16:51:42.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913790004 2020-09-08T16:51:42.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349 4-98784247865 5-335007449120 6-403726925851 7-158913790004' 2020-09-08T16:51:42.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:42.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:51:42.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=176093659186 2020-09-08T16:51:42.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 176093659186 2020-09-08T16:51:42.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349 4-98784247865 5-335007449120 6-403726925851 7-158913790004 8-176093659186' 2020-09-08T16:51:42.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:42.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2020-09-08T16:51:42.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=429496729622 2020-09-08T16:51:42.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 429496729622 2020-09-08T16:51:42.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349 4-98784247865 5-335007449120 6-403726925851 7-158913790004 8-176093659186 9-429496729622' 2020-09-08T16:51:42.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:51:42.468 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2020-09-08T16:51:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=219043332143 2020-09-08T16:51:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 219043332143 2020-09-08T16:51:42.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-60129542204 3-309237645349 4-98784247865 5-335007449120 6-403726925851 7-158913790004 8-176093659186 9-429496729622 10-219043332143' 2020-09-08T16:51:42.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:42.597 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836543 2020-09-08T16:51:42.597 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:42.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:51:42.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836543 2020-09-08T16:51:42.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:42.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836543 2020-09-08T16:51:42.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2020-09-08T16:51:42.599 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 21474836543 2020-09-08T16:51:42.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:51:43.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836543 -lt 21474836543 2020-09-08T16:51:43.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:43.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673021 2020-09-08T16:51:43.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:43.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:51:43.042 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673021 2020-09-08T16:51:43.042 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:43.042 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673021 2020-09-08T16:51:43.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673021' 2020-09-08T16:51:43.043 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 42949673021 2020-09-08T16:51:43.043 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:51:43.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673021 -lt 42949673021 2020-09-08T16:51:43.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:43.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542204 2020-09-08T16:51:43.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:43.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:51:43.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542204 2020-09-08T16:51:43.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:43.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542204 2020-09-08T16:51:43.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542204' 2020-09-08T16:51:43.419 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 60129542204 2020-09-08T16:51:43.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:51:43.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542204 -lt 60129542204 2020-09-08T16:51:43.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:43.838 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-309237645349 2020-09-08T16:51:43.838 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:43.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:51:43.840 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-309237645349 2020-09-08T16:51:43.840 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:43.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=309237645349 2020-09-08T16:51:43.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 309237645349' 2020-09-08T16:51:43.841 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 309237645349 2020-09-08T16:51:43.842 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:51:44.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 309237645350 -lt 309237645349 2020-09-08T16:51:44.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:44.209 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247865 2020-09-08T16:51:44.209 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:44.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:51:44.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247865 2020-09-08T16:51:44.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:44.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247865 2020-09-08T16:51:44.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247865' 2020-09-08T16:51:44.212 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 98784247865 2020-09-08T16:51:44.213 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:51:44.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247865 -lt 98784247865 2020-09-08T16:51:44.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:44.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-335007449120 2020-09-08T16:51:44.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:44.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:51:44.583 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-335007449120 2020-09-08T16:51:44.583 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:44.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=335007449120 2020-09-08T16:51:44.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 335007449120' 2020-09-08T16:51:44.584 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 335007449120 2020-09-08T16:51:44.585 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:51:44.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 335007449121 -lt 335007449120 2020-09-08T16:51:44.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:44.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-403726925851 2020-09-08T16:51:44.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:44.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:51:44.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-403726925851 2020-09-08T16:51:44.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:44.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=403726925851 2020-09-08T16:51:44.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 403726925851' 2020-09-08T16:51:44.960 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:51:44.960 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 403726925851 2020-09-08T16:51:45.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 403726925851 -lt 403726925851 2020-09-08T16:51:45.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:45.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-158913790004 2020-09-08T16:51:45.337 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:45.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:51:45.338 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-158913790004 2020-09-08T16:51:45.338 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:45.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913790004 2020-09-08T16:51:45.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 158913790004' 2020-09-08T16:51:45.339 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.7 seq 158913790004 2020-09-08T16:51:45.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:51:45.707 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913790004 -lt 158913790004 2020-09-08T16:51:45.707 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:45.708 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-176093659186 2020-09-08T16:51:45.708 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:45.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:51:45.709 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-176093659186 2020-09-08T16:51:45.710 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:45.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=176093659186 2020-09-08T16:51:45.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 176093659186' 2020-09-08T16:51:45.711 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.8 seq 176093659186 2020-09-08T16:51:45.711 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:51:46.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 176093659187 -lt 176093659186 2020-09-08T16:51:46.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:46.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-429496729622 2020-09-08T16:51:46.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:46.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2020-09-08T16:51:46.101 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-429496729622 2020-09-08T16:51:46.101 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:46.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=429496729622 2020-09-08T16:51:46.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 429496729622' 2020-09-08T16:51:46.102 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2020-09-08T16:51:46.102 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.9 seq 429496729622 2020-09-08T16:51:46.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 429496729623 -lt 429496729622 2020-09-08T16:51:46.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:51:46.472 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-219043332143 2020-09-08T16:51:46.472 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:51:46.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2020-09-08T16:51:46.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-219043332143 2020-09-08T16:51:46.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:51:46.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=219043332143 2020-09-08T16:51:46.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 219043332143' 2020-09-08T16:51:46.475 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.10 seq 219043332143 2020-09-08T16:51:46.476 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2020-09-08T16:51:46.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 219043332144 -lt 219043332143 2020-09-08T16:51:46.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:51:46.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:46.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:47.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 28 == 0 2020-09-08T16:51:47.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:51:47.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:51:47.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:51:47.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:51:47.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:51:47.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:51:47.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:51:47.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=25 2020-09-08T16:51:47.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:51:47.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:47.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:48.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 25 = 28 2020-09-08T16:51:48.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 25 '!=' -1 2020-09-08T16:51:48.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T16:51:48.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=25 2020-09-08T16:51:48.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:51:48.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:51:48.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:51:48.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:51:48.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:51:48.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:51:48.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:51:48.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:51:48.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:51:48.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:51:48.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=28 2020-09-08T16:51:48.699 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:51:48.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:51:48.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:51:49.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 28 = 28 2020-09-08T16:51:49.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:51:49.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:51:49.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2020-09-08T16:51:49.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:51:49.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:49.188 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2020-09-08T16:51:49.188 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:49.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=5 2020-09-08T16:51:49.561 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:51:49.561 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:51:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:51:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:10' 2020-09-08T16:51:49.562 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 5 1 9 4 3 10 2020-09-08T16:51:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2020-09-08T16:51:49.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:51:49.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:225: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2020-09-08T16:51:49.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=pool-jerasure 2020-09-08T16:51:49.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T16:51:50.053 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T16:51:50.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2020-09-08T16:51:50.983 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile profile-jerasure does not exist 2020-09-08T16:51:50.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2020-09-08T16:51:50.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2020-09-08T16:51:50.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2020-09-08T16:51:50.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:153: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2020-09-08T16:51:50.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:154: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2020-09-08T16:51:50.993 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2020-09-08T16:51:51.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:161: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2020-09-08T16:51:51.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2020-09-08T16:51:52.912 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-lrc-a' already exists 2020-09-08T16:51:52.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:51:53.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:164: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2020-09-08T16:51:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2020-09-08T16:51:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2020-09-08T16:51:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2020-09-08T16:51:53.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:53.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2020-09-08T16:51:53.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:53.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2020-09-08T16:51:53.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:53.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2020-09-08T16:51:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:51:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2020-09-08T16:51:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2020-09-08T16:51:55.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:51:55.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:51:55.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2020-09-08T16:51:55.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:51:55.281 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2020-09-08T16:51:55.281 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-lrc-a 2020-09-08T16:51:55.282 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:55.282 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2020-09-08T16:51:55.282 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:55.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=5 2020-09-08T16:51:55.699 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:55.699 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T16:51:55.699 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 5 9 3 2020-09-08T16:51:55.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2020-09-08T16:51:55.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2020-09-08T16:51:55.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2020-09-08T16:51:56.619 INFO:tasks.workunit.client.0.smithi120.stderr:osd.3 is already out. 2020-09-08T16:51:56.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-lrc-a SOMETHING 2020-09-08T16:51:56.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-lrc-a 2020-09-08T16:51:56.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:51:56.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<3\>' 2020-09-08T16:51:56.629 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2020-09-08T16:51:56.629 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:51:57.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=5 2020-09-08T16:51:57.117 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:51:57.117 INFO:tasks.workunit.client.0.smithi120.stderr:7' 2020-09-08T16:51:57.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 5 9 7 2020-09-08T16:51:57.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:51:59.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:51:59.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 3 2020-09-08T16:51:59.700 INFO:tasks.workunit.client.0.smithi120.stderr:osd.3 is already in. 2020-09-08T16:51:59.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:51:59.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:166: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2020-09-08T16:51:59.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=pool-lrc-a 2020-09-08T16:51:59.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2020-09-08T16:52:00.915 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-lrc-a' does not exist 2020-09-08T16:52:00.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:167: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2020-09-08T16:52:01.761 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile profile-lrc-a does not exist 2020-09-08T16:52:01.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2020-09-08T16:52:01.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2020-09-08T16:52:01.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2020-09-08T16:52:01.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:172: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2020-09-08T16:52:01.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:173: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2020-09-08T16:52:01.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2020-09-08T16:52:02.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2020-09-08T16:52:02.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2020-09-08T16:52:03.656 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-lrc' already exists 2020-09-08T16:52:03.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:52:04.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:182: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2020-09-08T16:52:04.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2020-09-08T16:52:04.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2020-09-08T16:52:04.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2020-09-08T16:52:04.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:04.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2020-09-08T16:52:04.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:04.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2020-09-08T16:52:04.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:04.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2020-09-08T16:52:04.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:04.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2020-09-08T16:52:04.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2020-09-08T16:52:05.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:52:06.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:52:06.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2020-09-08T16:52:06.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:52:06.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2020-09-08T16:52:06.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-lrc 2020-09-08T16:52:06.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:52:06.165 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-lrc SOMETHING 2020-09-08T16:52:06.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:52:06.794 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=8 2020-09-08T16:52:06.795 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:52:06.795 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:52:06.795 INFO:tasks.workunit.client.0.smithi120.stderr:10 2020-09-08T16:52:06.795 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:52:06.796 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:52:06.796 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:52:06.796 INFO:tasks.workunit.client.0.smithi120.stderr:7' 2020-09-08T16:52:06.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 8 6 0 10 5 3 9 7 2020-09-08T16:52:06.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2020-09-08T16:52:06.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2020-09-08T16:52:06.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2020-09-08T16:52:07.637 INFO:tasks.workunit.client.0.smithi120.stderr:osd.7 is already out. 2020-09-08T16:52:07.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-lrc SOMETHING 2020-09-08T16:52:07.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<7\>' 2020-09-08T16:52:07.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-lrc 2020-09-08T16:52:07.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:52:07.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-lrc SOMETHING 2020-09-08T16:52:07.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:52:08.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=8 2020-09-08T16:52:08.258 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:52:08.258 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:52:08.258 INFO:tasks.workunit.client.0.smithi120.stderr:10 2020-09-08T16:52:08.259 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:52:08.259 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:52:08.259 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:52:08.259 INFO:tasks.workunit.client.0.smithi120.stderr:4' 2020-09-08T16:52:08.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 8 6 0 10 5 3 9 4 2020-09-08T16:52:08.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:52:09.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:52:09.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 7 2020-09-08T16:52:10.641 INFO:tasks.workunit.client.0.smithi120.stderr:osd.7 is already in. 2020-09-08T16:52:10.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:52:10.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:184: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2020-09-08T16:52:10.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=pool-lrc 2020-09-08T16:52:10.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2020-09-08T16:52:12.362 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-lrc' does not exist 2020-09-08T16:52:12.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:185: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2020-09-08T16:52:13.597 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile profile-lrc does not exist 2020-09-08T16:52:13.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2020-09-08T16:52:13.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2020-09-08T16:52:13.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2020-09-08T16:52:13.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:232: TEST_rados_put_get_shec: local poolname=pool-shec 2020-09-08T16:52:13.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:233: TEST_rados_put_get_shec: local profile=profile-shec 2020-09-08T16:52:13.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:235: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2020-09-08T16:52:14.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2020-09-08T16:52:14.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2020-09-08T16:52:15.594 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-shec' already exists 2020-09-08T16:52:15.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:52:16.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:242: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2020-09-08T16:52:16.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2020-09-08T16:52:16.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2020-09-08T16:52:16.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2020-09-08T16:52:16.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:16.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2020-09-08T16:52:16.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:16.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2020-09-08T16:52:16.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:16.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2020-09-08T16:52:16.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2020-09-08T16:52:16.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2020-09-08T16:52:16.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2020-09-08T16:52:17.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:52:18.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:52:18.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2020-09-08T16:52:18.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:52:18.341 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2020-09-08T16:52:18.341 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-shec 2020-09-08T16:52:18.341 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:52:18.342 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-shec SOMETHING 2020-09-08T16:52:18.342 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:52:19.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=7 2020-09-08T16:52:19.006 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:52:19.006 INFO:tasks.workunit.client.0.smithi120.stderr:1' 2020-09-08T16:52:19.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 7 9 1 2020-09-08T16:52:19.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2020-09-08T16:52:19.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2020-09-08T16:52:19.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2020-09-08T16:52:19.721 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 is already out. 2020-09-08T16:52:19.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-shec SOMETHING 2020-09-08T16:52:19.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<1\>' 2020-09-08T16:52:19.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-shec 2020-09-08T16:52:19.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=SOMETHING 2020-09-08T16:52:19.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-shec SOMETHING 2020-09-08T16:52:19.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:52:20.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=7 2020-09-08T16:52:20.539 INFO:tasks.workunit.client.0.smithi120.stderr:9 2020-09-08T16:52:20.539 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T16:52:20.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 7 9 6 2020-09-08T16:52:20.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2020-09-08T16:52:22.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2020-09-08T16:52:22.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 1 2020-09-08T16:52:22.913 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 is already in. 2020-09-08T16:52:22.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2020-09-08T16:52:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:244: TEST_rados_put_get_shec: delete_pool pool-shec 2020-09-08T16:52:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=pool-shec 2020-09-08T16:52:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2020-09-08T16:52:24.216 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-shec' does not exist 2020-09-08T16:52:24.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:245: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2020-09-08T16:52:25.239 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile profile-shec does not exist 2020-09-08T16:52:25.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2020-09-08T16:52:25.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=ecpool 2020-09-08T16:52:25.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2020-09-08T16:52:26.059 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'ecpool' does not exist 2020-09-08T16:52:26.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2020-09-08T16:52:26.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2020-09-08T16:52:26.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:52:26.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2020-09-08T16:52:26.069 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:52:26.070 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:52:26.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:52:26.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:52:26.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:52:26.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:52:26.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:52:26.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:52:26.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:52:26.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:52:26.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:52:26.192 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:52:26.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:52:26.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:52:26.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:52:26.194 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:52:26.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:52:26.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:52:26.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2020-09-08T16:52:26.240 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:52:26.241 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:26.241 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:52:26.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35290 2020-09-08T16:52:26.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:52:26.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:52:26.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2020-09-08T16:52:26.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/test-erasure-code 0 2020-09-08T16:52:26.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2020-09-08T16:52:26.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-09-08T16:52:26.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2020-09-08T16:52:26.244 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:52:26.245 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:52:26.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:52:26.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:52:26.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:52:26.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:52:26.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:52:26.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:52:26.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:52:26.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:52:26.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:52:26.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:52:26.249 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:52:26.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:52:26.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:52:26.250 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:52:26.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:52:26.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-09-08T16:52:26.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2020-09-08T16:52:26.254 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:52:26.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:26.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35290 2020-09-08T16:52:26.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35290 2020-09-08T16:52:26.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:52:26.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:52:26.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2020-09-08T16:52:26.257 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-09-08T16:52:26.258 INFO:teuthology.orchestra.run.smithi120:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-09-08T16:52:26.329 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2020-09-08T16:52:26.330 INFO:teuthology.orchestra.run.smithi120:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dce4d291bd4243c9802c17d67b85a6da6bbc0f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2020-09-08T16:52:26.389 INFO:tasks.workunit.client.0.smithi120.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-09-08T16:52:26.398 INFO:tasks.workunit.client.0.smithi120.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-09-08T16:52:26.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-09-08T16:52:26.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-09-08T16:52:26.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2020-09-08T16:52:26.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2020-09-08T16:52:26.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2020-09-08T16:52:26.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2020-09-08T16:52:26.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2020-09-08T16:52:26.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2020-09-08T16:52:26.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/test-erasure-eio 2020-09-08T16:52:26.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2020-09-08T16:52:26.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2020-09-08T16:52:26.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2020-09-08T16:52:26.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2020-09-08T16:52:26.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2020-09-08T16:52:26.401 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2020-09-08T16:52:26.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none ' 2020-09-08T16:52:26.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2020-09-08T16:52:26.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:30: run: set 2020-09-08T16:52:26.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-09-08T16:52:26.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:30: run: local 'funcs=TEST_ec_backfill_unfound 2020-09-08T16:52:26.405 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_ec_object_attr_read_error 2020-09-08T16:52:26.405 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_ec_recovery_multiple_errors 2020-09-08T16:52:26.405 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_ec_recovery_multiple_objects 2020-09-08T16:52:26.405 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_ec_recovery_multiple_objects_eio 2020-09-08T16:52:26.405 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_ec_recovery_unfound 2020-09-08T16:52:26.406 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_ec_single_recovery_error 2020-09-08T16:52:26.406 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_get_bad_size_shard_0 2020-09-08T16:52:26.406 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_get_bad_size_shard_1 2020-09-08T16:52:26.406 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_get_subread_eio_shard_0 2020-09-08T16:52:26.406 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_get_subread_eio_shard_1 2020-09-08T16:52:26.407 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_get_subread_missing 2020-09-08T16:52:26.407 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2020-09-08T16:52:26.407 INFO:tasks.workunit.client.0.smithi120.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2020-09-08T16:52:26.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T16:52:26.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T16:52:26.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T16:52:26.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T16:52:26.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T16:52:26.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:52:26.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T16:52:26.409 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:52:26.409 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:52:26.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:52:26.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:52:26.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:52:26.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:52:26.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:52:26.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:52:26.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:52:26.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:52:26.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:52:26.411 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:52:26.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:52:26.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:52:26.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:52:26.414 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:52:26.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:52:26.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:52:26.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T16:52:26.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:52:26.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:26.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:26.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T16:52:26.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:52:26.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:52:26.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T16:52:26.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:52:26.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:26.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:26.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T16:52:26.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:52:26.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-09-08T16:52:26.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-09-08T16:52:26.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:52:26.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T16:52:26.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T16:52:26.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T16:52:26.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:52:26.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:52:26.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:52:26.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T16:52:26.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T16:52:26.709 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:52:26.709 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:52:26.709 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:52:26.709 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:52:26.710 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:26.710 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:26.710 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:26.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T16:52:26.885 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:52:26.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:52:26.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:52:26.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:52:26.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:52:26.887 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:52:26.888 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:52:26.888 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:52:26.888 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:52:26.888 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:52:26.888 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:26.889 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:26.889 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:52:26.889 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:52:26.889 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T16:52:26.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:52:26.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:52:26.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:52:26.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:52:26.975 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:52:26.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:52:26.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:52:26.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:52:26.975 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:52:26.976 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:26.976 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:26.976 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:52:26.976 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:52:26.976 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T16:52:27.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T16:52:27.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T16:52:27.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:52:27.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:52:27.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:52:27.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T16:52:27.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:52:27.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:52:27.974 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:52:27.975 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:52:27.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:52:27.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:27.975 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:27.975 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:27.976 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:52:27.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:52:27.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T16:52:27.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T16:52:28.329 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T16:52:28.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:52:29.339 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T16:52:29.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:52:29.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:52:29.340 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:52:29.340 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:29.340 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:29.341 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:52:29.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T16:52:29.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T16:52:29.421 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:52:29.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T16:52:29.431 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:52:26.741+0000 7fa29272b6c0 0 load: jerasure load: lrc load: isa 2020-09-08T16:52:29.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_ec_backfill_unfound td/test-erasure-eio 2020-09-08T16:52:29.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:510: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2020-09-08T16:52:29.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local objname=myobject 2020-09-08T16:52:29.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local lastobj=300 2020-09-08T16:52:29.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:514: TEST_ec_backfill_unfound: local testobj=obj250 2020-09-08T16:52:29.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:516: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:52:29.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T16:52:29.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: setup_osds 5 2020-09-08T16:52:29.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=5 2020-09-08T16:52:29.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T16:52:29.433 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 5 - 1 2020-09-08T16:52:29.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 4 2020-09-08T16:52:29.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:52:29.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T16:52:29.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:52:29.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:52:29.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:52:29.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:52:29.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T16:52:29.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T16:52:29.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:52:29.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:52:29.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:52:29.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T16:52:29.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T16:52:29.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:52:29.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:52:29.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:52:29.441 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:52:29.441 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:52:29.442 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:52:29.442 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:52:29.442 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:29.442 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:29.442 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:29.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:29.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:52:29.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:52:29.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:52:29.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:52:29.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:52:29.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:52:29.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:52:29.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:52:29.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:52:29.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:52:29.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T16:52:29.445 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:52:29.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c16dcc0a-5c52-4e5c-8089-92c7d07f26a4 2020-09-08T16:52:29.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c16dcc0a-5c52-4e5c-8089-92c7d07f26a4' 2020-09-08T16:52:29.445 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:52:29.446 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 c16dcc0a-5c52-4e5c-8089-92c7d07f26a4 2020-09-08T16:52:29.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDNtldfI74sGxAANAcTPQbzha86JNdYivuBpQ== 2020-09-08T16:52:29.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDNtldfI74sGxAANAcTPQbzha86JNdYivuBpQ=="}' 2020-09-08T16:52:29.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c16dcc0a-5c52-4e5c-8089-92c7d07f26a4 -i td/test-erasure-eio/0/new.json 2020-09-08T16:52:29.785 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:52:29.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T16:52:29.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDNtldfI74sGxAANAcTPQbzha86JNdYivuBpQ== --osd-uuid c16dcc0a-5c52-4e5c-8089-92c7d07f26a4 2020-09-08T16:52:29.821 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:29.826+0000 7f13a7fc5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:29.822 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:29.828+0000 7f13a7fc5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:29.824 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:29.829+0000 7f13a7fc5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:29.825 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:29.830+0000 7f13a7fc5ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:52:29.825 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:29.830+0000 7f13a7fc5ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:52:29.825 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:29.830+0000 7f13a7fc5ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:52:29.871 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:29.876+0000 7f13a7fc5ec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T16:52:34.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T16:52:34.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:52:34.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:52:34.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:52:34.547 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:52:35.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:52:35.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:52:35.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:52:35.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:52:35.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:52:35.048 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:52:35.069 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:35.074+0000 7fb35b579ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:35.071 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:35.075+0000 7fb35b579ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:35.072 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:35.077+0000 7fb35b579ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:35.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:52:35.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:52:35.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:52:35.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:52:35.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:52:35.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:52:35.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:52:35.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:52:35.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:52:35.417 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:52:35.589 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:35.591+0000 7fb35b579ec0 -1 Falling back to public interface 2020-09-08T16:52:35.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:52:36.368 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:36.373+0000 7fb35b579ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:52:36.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:52:36.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:52:36.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:52:36.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:52:36.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:52:36.791 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:52:37.215 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/60047,v1:127.0.0.1:6803/60047] [v2:127.0.0.1:6804/60047,v1:127.0.0.1:6805/60047] exists,up c16dcc0a-5c52-4e5c-8089-92c7d07f26a4 2020-09-08T16:52:37.216 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 324ca0d6-1991-43b0-b4f2-d7c0a8795587 2020-09-08T16:52:37.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:52:37.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:52:37.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:52:37.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:52:37.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T16:52:37.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:52:37.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:52:37.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T16:52:37.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:52:37.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T16:52:37.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T16:52:37.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:52:37.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:52:37.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:52:37.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T16:52:37.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T16:52:37.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:52:37.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:52:37.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:52:37.220 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:52:37.220 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:52:37.220 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:52:37.221 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:52:37.221 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:37.221 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:37.221 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:37.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:37.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:52:37.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:52:37.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:52:37.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:52:37.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:52:37.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:52:37.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:52:37.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:52:37.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:52:37.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:52:37.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T16:52:37.224 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:52:37.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=324ca0d6-1991-43b0-b4f2-d7c0a8795587 2020-09-08T16:52:37.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 324ca0d6-1991-43b0-b4f2-d7c0a8795587' 2020-09-08T16:52:37.224 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:52:37.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDVtldfX3sMDhAAI4R/GwocSsK2XfMQ9mHqug== 2020-09-08T16:52:37.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDVtldfX3sMDhAAI4R/GwocSsK2XfMQ9mHqug=="}' 2020-09-08T16:52:37.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 324ca0d6-1991-43b0-b4f2-d7c0a8795587 -i td/test-erasure-eio/1/new.json 2020-09-08T16:52:37.896 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:52:37.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T16:52:37.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDVtldfX3sMDhAAI4R/GwocSsK2XfMQ9mHqug== --osd-uuid 324ca0d6-1991-43b0-b4f2-d7c0a8795587 2020-09-08T16:52:37.931 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:37.936+0000 7fd4b87acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:37.933 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:37.938+0000 7fd4b87acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:37.934 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:37.939+0000 7fd4b87acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:37.935 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:37.940+0000 7fd4b87acec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:52:37.935 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:37.940+0000 7fd4b87acec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:52:37.936 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:37.940+0000 7fd4b87acec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:52:38.038 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:38.043+0000 7fd4b87acec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T16:52:42.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T16:52:42.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:52:42.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T16:52:42.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:52:42.461 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T16:52:42.955 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T16:52:42.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T16:52:42.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:52:42.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:52:42.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:52:42.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:52:42.976 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:42.980+0000 7f70c214dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:42.981 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:42.982+0000 7f70c214dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:42.981 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:42.984+0000 7f70c214dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:43.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T16:52:43.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:52:43.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T16:52:43.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:52:43.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:52:43.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:52:43.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:52:43.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:52:43.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:52:43.322 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:52:43.496 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:43.499+0000 7f70c214dec0 -1 Falling back to public interface 2020-09-08T16:52:43.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:52:44.353 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:44.357+0000 7f70c214dec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T16:52:44.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:52:44.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:52:44.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:52:44.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:52:44.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:52:44.706 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:52:45.142 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/61041,v1:127.0.0.1:6811/61041] [v2:127.0.0.1:6812/61041,v1:127.0.0.1:6813/61041] exists,up 324ca0d6-1991-43b0-b4f2-d7c0a8795587 2020-09-08T16:52:45.142 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 436ae6f4-f6e2-48c6-8f23-0ec18c0322ce 2020-09-08T16:52:45.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:52:45.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:52:45.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:52:45.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:52:45.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T16:52:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:52:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:52:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T16:52:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:52:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T16:52:45.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T16:52:45.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:52:45.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:52:45.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:52:45.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T16:52:45.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T16:52:45.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:52:45.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:52:45.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:52:45.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:52:45.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:52:45.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:52:45.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:52:45.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:45.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:45.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:45.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:45.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:52:45.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:52:45.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:52:45.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:52:45.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:52:45.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:52:45.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:52:45.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:52:45.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:52:45.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:52:45.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T16:52:45.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:52:45.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=436ae6f4-f6e2-48c6-8f23-0ec18c0322ce 2020-09-08T16:52:45.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 436ae6f4-f6e2-48c6-8f23-0ec18c0322ce' 2020-09-08T16:52:45.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:52:45.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDdtldfjDS2CRAAyGVPgRc4iFaZddjlW9HmwQ== 2020-09-08T16:52:45.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDdtldfjDS2CRAAyGVPgRc4iFaZddjlW9HmwQ=="}' 2020-09-08T16:52:45.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 436ae6f4-f6e2-48c6-8f23-0ec18c0322ce -i td/test-erasure-eio/2/new.json 2020-09-08T16:52:45.978 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:52:45.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T16:52:45.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdtldfjDS2CRAAyGVPgRc4iFaZddjlW9HmwQ== --osd-uuid 436ae6f4-f6e2-48c6-8f23-0ec18c0322ce 2020-09-08T16:52:46.015 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:46.019+0000 7fc5b22d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:46.016 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:46.020+0000 7fc5b22d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:46.017 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:46.022+0000 7fc5b22d9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:46.018 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:46.023+0000 7fc5b22d9ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:52:46.018 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:46.023+0000 7fc5b22d9ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:52:46.019 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:46.023+0000 7fc5b22d9ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:52:46.075 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:46.080+0000 7fc5b22d9ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T16:52:50.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T16:52:50.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:52:50.976 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T16:52:50.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T16:52:50.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:52:51.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T16:52:51.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:52:51.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:52:51.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:52:51.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:52:51.474 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T16:52:51.495 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:51.500+0000 7f4db4fc9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:51.522 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:51.527+0000 7f4db4fc9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:51.524 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:51.528+0000 7f4db4fc9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:51.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T16:52:51.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:52:51.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T16:52:51.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:52:51.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:52:51.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:52:51.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:52:51.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:52:51.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:52:51.849 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:52:52.040 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:52.044+0000 7f4db4fc9ec0 -1 Falling back to public interface 2020-09-08T16:52:52.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:52:52.859 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:52.863+0000 7f4db4fc9ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T16:52:53.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:52:53.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:52:53.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:52:53.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:52:53.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:52:53.222 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:52:53.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:52:54.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:52:54.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:52:54.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:52:54.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:52:54.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:52:54.627 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:52:55.110 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 18 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/62035,v1:127.0.0.1:6819/62035] [v2:127.0.0.1:6820/62035,v1:127.0.0.1:6821/62035] exists,up 436ae6f4-f6e2-48c6-8f23-0ec18c0322ce 2020-09-08T16:52:55.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:52:55.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:52:55.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:52:55.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:52:55.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T16:52:55.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:52:55.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:52:55.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T16:52:55.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:52:55.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T16:52:55.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T16:52:55.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:52:55.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:52:55.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:52:55.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T16:52:55.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T16:52:55.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:52:55.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:52:55.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:52:55.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:52:55.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:52:55.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:52:55.115 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:52:55.116 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:52:55.116 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:52:55.116 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:55.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:52:55.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:52:55.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:52:55.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:52:55.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:52:55.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:52:55.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:52:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:52:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:52:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:52:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:52:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T16:52:55.119 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:52:55.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=575723af-2b73-4e33-bc9c-c5f5159d9c19 2020-09-08T16:52:55.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 575723af-2b73-4e33-bc9c-c5f5159d9c19' 2020-09-08T16:52:55.120 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:52:55.120 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 575723af-2b73-4e33-bc9c-c5f5159d9c19 2020-09-08T16:52:55.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDntldfQongBxAAHUO2YBEAy4aYnxvktRKXfA== 2020-09-08T16:52:55.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDntldfQongBxAAHUO2YBEAy4aYnxvktRKXfA=="}' 2020-09-08T16:52:55.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 575723af-2b73-4e33-bc9c-c5f5159d9c19 -i td/test-erasure-eio/3/new.json 2020-09-08T16:52:55.726 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T16:52:55.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T16:52:55.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDntldfQongBxAAHUO2YBEAy4aYnxvktRKXfA== --osd-uuid 575723af-2b73-4e33-bc9c-c5f5159d9c19 2020-09-08T16:52:55.761 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:55.765+0000 7f1a5328dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:55.763 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:55.767+0000 7f1a5328dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:55.770 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:55.768+0000 7f1a5328dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:52:55.770 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:55.769+0000 7f1a5328dec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:52:55.771 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:55.769+0000 7f1a5328dec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:52:55.771 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:55.769+0000 7f1a5328dec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:52:55.894 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:52:55.899+0000 7f1a5328dec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T16:53:00.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T16:53:00.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:53:00.605 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T16:53:00.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T16:53:00.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:53:01.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T16:53:01.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:53:01.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:53:01.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:53:01.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:53:01.106 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T16:53:01.127 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:01.131+0000 7f896b7edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:01.129 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:01.133+0000 7f896b7edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:01.129 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:01.134+0000 7f896b7edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:01.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T16:53:01.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:53:01.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T16:53:01.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:53:01.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:53:01.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:53:01.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:53:01.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:53:01.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:53:01.498 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:53:01.645 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:01.650+0000 7f896b7edec0 -1 Falling back to public interface 2020-09-08T16:53:01.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:53:02.461 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:02.465+0000 7f896b7edec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T16:53:02.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:53:02.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:53:02.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:53:02.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:53:02.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:53:02.863 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:53:03.365 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/63055,v1:127.0.0.1:6827/63055] [v2:127.0.0.1:6828/63055,v1:127.0.0.1:6829/63055] exists,up 575723af-2b73-4e33-bc9c-c5f5159d9c19 2020-09-08T16:53:03.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:53:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:53:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:53:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:53:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 4 2020-09-08T16:53:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:53:03.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:53:03.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T16:53:03.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:53:03.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/4 2020-09-08T16:53:03.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T16:53:03.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:53:03.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:53:03.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:53:03.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2020-09-08T16:53:03.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2020-09-08T16:53:03.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:53:03.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:53:03.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:53:03.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:53:03.418 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:53:03.419 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:53:03.419 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:53:03.419 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:53:03.420 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:53:03.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:53:03.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:53:03.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:53:03.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:53:03.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:53:03.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:53:03.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:53:03.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:53:03.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:53:03.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:53:03.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:53:03.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:53:03.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/4 2020-09-08T16:53:03.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:53:03.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=bfcf943a-1711-4dec-a105-3998e40e6b0a 2020-09-08T16:53:03.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 bfcf943a-1711-4dec-a105-3998e40e6b0a' 2020-09-08T16:53:03.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:53:03.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDvtldfU9sdFxAAhZqpD6wF28t9OMzgwasU5g== 2020-09-08T16:53:03.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDvtldfU9sdFxAAhZqpD6wF28t9OMzgwasU5g=="}' 2020-09-08T16:53:03.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new bfcf943a-1711-4dec-a105-3998e40e6b0a -i td/test-erasure-eio/4/new.json 2020-09-08T16:53:03.425 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 bfcf943a-1711-4dec-a105-3998e40e6b0a 2020-09-08T16:53:04.154 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T16:53:04.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/4/new.json 2020-09-08T16:53:04.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDvtldfU9sdFxAAhZqpD6wF28t9OMzgwasU5g== --osd-uuid bfcf943a-1711-4dec-a105-3998e40e6b0a 2020-09-08T16:53:04.188 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:04.194+0000 7f8829aeaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:04.190 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:04.196+0000 7f8829aeaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:04.192 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:04.197+0000 7f8829aeaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:04.192 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:04.198+0000 7f8829aeaec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:53:04.192 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:04.198+0000 7f8829aeaec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:53:04.193 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:04.198+0000 7f8829aeaec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:53:04.340 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:04.345+0000 7f8829aeaec0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2020-09-08T16:53:09.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2020-09-08T16:53:09.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:53:09.197 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T16:53:09.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T16:53:09.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:53:09.697 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T16:53:09.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T16:53:09.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:53:09.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:53:09.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:53:09.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:53:09.720 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:09.725+0000 7f5d8e735ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:09.721 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:09.727+0000 7f5d8e735ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:09.730 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:09.728+0000 7f5d8e735ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:53:10.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T16:53:10.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:53:10.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T16:53:10.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:53:10.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:53:10.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:53:10.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:53:10.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:53:10.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:53:10.155 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:53:10.240 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:10.246+0000 7f5d8e735ec0 -1 Falling back to public interface 2020-09-08T16:53:10.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:53:11.090 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:53:11.092+0000 7f5d8e735ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T16:53:11.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:53:11.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:53:11.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:53:11.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:53:11.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:53:11.528 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:53:11.911 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64049,v1:127.0.0.1:6835/64049] [v2:127.0.0.1:6836/64049,v1:127.0.0.1:6837/64049] exists,up bfcf943a-1711-4dec-a105-3998e40e6b0a 2020-09-08T16:53:11.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:53:11.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:53:11.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:53:11.914 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T16:53:11.915 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:53:11.915 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:53:11.916 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:53:11.916 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:53:11.916 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:53:11.917 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T16:53:11.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T16:53:11.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T16:53:12.003 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:53:12.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T16:53:12.012 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:52:35.600+0000 7fb35b579ec0 0 load: jerasure load: lrc load: isa 2020-09-08T16:53:12.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:53:12.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:521: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2020-09-08T16:53:12.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2020-09-08T16:53:12.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T16:53:12.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T16:53:12.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=3 2020-09-08T16:53:12.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T16:53:12.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=2 2020-09-08T16:53:12.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T16:53:12.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2020-09-08T16:53:12.910 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T16:53:12.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T16:53:13.798 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T16:53:13.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:53:14.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T16:53:14.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:53:14.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:53:14.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:53:14.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:53:14.810 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:53:14.810 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:53:14.810 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:53:14.810 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:53:14.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:53:14.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:53:14.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:53:14.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:53:14.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:53:14.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:53:14.913 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:53:15.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:53:15.295 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:53:15.295 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:53:15.295 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:53:15.295 INFO:tasks.workunit.client.0.smithi120.stderr:4' 2020-09-08T16:53:15.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:53:15.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:53:15.296 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:53:15.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803786 2020-09-08T16:53:15.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803786 2020-09-08T16:53:15.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786' 2020-09-08T16:53:15.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:53:15.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:53:15.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607560 2020-09-08T16:53:15.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607560 2020-09-08T16:53:15.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786 1-51539607560' 2020-09-08T16:53:15.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:53:15.537 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:53:15.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411336 2020-09-08T16:53:15.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411336 2020-09-08T16:53:15.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-77309411336' 2020-09-08T16:53:15.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:53:15.661 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:53:15.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964116997 2020-09-08T16:53:15.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964116997 2020-09-08T16:53:15.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-77309411336 3-115964116997' 2020-09-08T16:53:15.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:53:15.781 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:53:15.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=137438953475 2020-09-08T16:53:15.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 137438953475 2020-09-08T16:53:15.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786 1-51539607560 2-77309411336 3-115964116997 4-137438953475' 2020-09-08T16:53:15.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:53:15.903 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803786 2020-09-08T16:53:15.903 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:53:15.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:53:15.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803786 2020-09-08T16:53:15.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:53:15.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803786 2020-09-08T16:53:15.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803786' 2020-09-08T16:53:15.906 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803786 2020-09-08T16:53:15.906 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:53:16.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803786 -lt 25769803786 2020-09-08T16:53:16.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:53:16.272 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607560 2020-09-08T16:53:16.272 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:53:16.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:53:16.273 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607560 2020-09-08T16:53:16.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:53:16.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607560 2020-09-08T16:53:16.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2020-09-08T16:53:16.275 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 51539607560 2020-09-08T16:53:16.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:53:16.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607558 -lt 51539607560 2020-09-08T16:53:16.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:53:17.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T16:53:17.645 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:53:18.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607561 -lt 51539607560 2020-09-08T16:53:18.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:53:18.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411336 2020-09-08T16:53:18.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:53:18.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:53:18.019 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411336 2020-09-08T16:53:18.019 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:53:18.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411336 2020-09-08T16:53:18.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411336' 2020-09-08T16:53:18.020 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411336 2020-09-08T16:53:18.020 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:53:18.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411336 -lt 77309411336 2020-09-08T16:53:18.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:53:18.385 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964116997 2020-09-08T16:53:18.385 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:53:18.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:53:18.387 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964116997 2020-09-08T16:53:18.387 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:53:18.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964116997 2020-09-08T16:53:18.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964116997' 2020-09-08T16:53:18.389 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 115964116997 2020-09-08T16:53:18.389 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:53:18.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964116997 -lt 115964116997 2020-09-08T16:53:18.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:53:18.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-137438953475 2020-09-08T16:53:18.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:53:18.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:53:18.781 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-137438953475 2020-09-08T16:53:18.782 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:53:18.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=137438953475 2020-09-08T16:53:18.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 137438953475' 2020-09-08T16:53:18.783 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:53:18.783 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 137438953475 2020-09-08T16:53:19.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 137438953475 -lt 137438953475 2020-09-08T16:53:19.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:53:19.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:53:19.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:53:19.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T16:53:19.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:53:19.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:53:19.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:53:19.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:53:19.638 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:53:19.638 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:53:19.638 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:53:19.989 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T16:53:19.990 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:53:19.991 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:53:19.991 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:53:20.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T16:53:20.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:53:20.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:53:20.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:524: TEST_ec_backfill_unfound: ceph pg dump pgs 2020-09-08T16:53:20.831 INFO:tasks.workunit.client.0.smithi120.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-09-08T16:53:20.831 INFO:tasks.workunit.client.0.smithi120.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:15.834930+0000 0'0 38:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2020-09-08T16:53:13.398142+0000 0'0 2020-09-08T16:53:13.398142+0000 0 2020-09-08T16:53:20.831 INFO:tasks.workunit.client.0.smithi120.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:12.903984+0000 0'0 38:15 [4,1,2] 4 [4,1,2] 4 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:20.831 INFO:tasks.workunit.client.0.smithi120.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:13.742476+0000 0'0 38:63 [0,4,1] 0 [0,4,1] 0 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:20.832 INFO:tasks.workunit.client.0.smithi120.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:52:54.851372+0000 0'0 37:32 [1,0,2] 1 [1,0,2] 1 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:20.832 INFO:tasks.workunit.client.0.smithi120.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:12.870222+0000 0'0 38:15 [4,3,0] 4 [4,3,0] 4 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:20.832 INFO:tasks.workunit.client.0.smithi120.stdout: 2020-09-08T16:53:20.832 INFO:tasks.workunit.client.0.smithi120.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-09-08T16:53:20.833 INFO:tasks.workunit.client.0.smithi120.stderr:dumped pgs 2020-09-08T16:53:20.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:526: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2020-09-08T16:53:20.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T16:53:20.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T16:53:20.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=myobject 2020-09-08T16:53:20.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:53:20.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T16:53:20.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:53:20.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T16:53:20.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:53:20.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T16:53:20.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:53:20.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T16:53:20.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:21.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:53:21.057 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2020-09-08T16:53:21.058 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:53:21.058 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T16:53:21.058 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T16:53:21.058 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:53:21.421 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:53:21.422 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:53:21.422 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:53:21.422 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:53:21.422 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T16:53:21.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 4 0 2 2020-09-08T16:53:21.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local -a initial_osds 2020-09-08T16:53:21.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:529: TEST_ec_backfill_unfound: local last_osd=2 2020-09-08T16:53:21.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2020-09-08T16:53:21.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:53:21.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:53:21.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:53:21.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:53:21.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:53:21.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:53:21.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: ceph osd down 2 2020-09-08T16:53:22.073 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 is already down. 2020-09-08T16:53:22.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: ceph osd out 2 2020-09-08T16:53:22.803 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 is already out. 2020-09-08T16:53:22.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph pg dump pgs 2020-09-08T16:53:23.206 INFO:tasks.workunit.client.0.smithi120.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-09-08T16:53:23.206 INFO:tasks.workunit.client.0.smithi120.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:15.834930+0000 0'0 38:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2020-09-08T16:53:13.398142+0000 0'0 2020-09-08T16:53:13.398142+0000 0 2020-09-08T16:53:23.207 INFO:tasks.workunit.client.0.smithi120.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+undersized 2020-09-08T16:53:22.453192+0000 0'0 41:23 [4,1] 4 [4,1] 4 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:23.207 INFO:tasks.workunit.client.0.smithi120.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:13.742476+0000 0'0 38:63 [0,4,1] 0 [0,4,1] 0 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:23.207 INFO:tasks.workunit.client.0.smithi120.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:52:54.851372+0000 0'0 38:33 [1,0,2] 1 [1,0,2] 1 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:23.207 INFO:tasks.workunit.client.0.smithi120.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:12.870222+0000 0'0 40:17 [4,3,0] 4 [4,3,0] 4 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:53:23.208 INFO:tasks.workunit.client.0.smithi120.stdout: 2020-09-08T16:53:23.208 INFO:tasks.workunit.client.0.smithi120.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-09-08T16:53:23.208 INFO:tasks.workunit.client.0.smithi120.stderr:dumped pgs 2020-09-08T16:53:23.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2020-09-08T16:53:23.217 INFO:tasks.workunit.client.0.smithi120.stderr:4+0 records in 2020-09-08T16:53:23.217 INFO:tasks.workunit.client.0.smithi120.stderr:4+0 records out 2020-09-08T16:53:23.217 INFO:tasks.workunit.client.0.smithi120.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.5679e-05 s, 47.8 MB/s 2020-09-08T16:53:23.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: seq 1 300 2020-09-08T16:53:23.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:23.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:23.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:23.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:24.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:24.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:24.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:24.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:24.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:24.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:24.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:24.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:25.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:25.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:25.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:25.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:25.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:25.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:25.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:25.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:26.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:26.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:26.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:26.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:26.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:26.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:26.880 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:26.880 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:27.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:27.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:27.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:27.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:27.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:27.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:27.911 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:27.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:28.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:28.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:28.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:28.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:28.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:28.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:28.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:28.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:29.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:29.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:29.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:29.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:29.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:29.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:29.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:29.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:30.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:30.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:30.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:30.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:30.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:30.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:31.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:31.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:31.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:31.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:31.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:31.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:31.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:31.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:32.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:32.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:32.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:32.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:32.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:32.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:32.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:32.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:33.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:33.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:33.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:33.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:33.681 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:33.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:33.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:33.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:34.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:34.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:34.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:34.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:34.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:34.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:34.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:34.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:35.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:35.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:35.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:35.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:35.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:35.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:35.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:35.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:36.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:36.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:36.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:36.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:36.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:36.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:36.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:36.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:37.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:37.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:37.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:37.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:37.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:37.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:37.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:37.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:38.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:38.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:38.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:38.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:38.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:38.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:38.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:38.984 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:39.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:39.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:39.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:39.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:39.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:39.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:40.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:40.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:40.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:40.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:40.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:40.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:40.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:40.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:41.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:41.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:41.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:41.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:41.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:41.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:41.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:41.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:42.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:42.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:42.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:42.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:42.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:42.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:42.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:42.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:43.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:43.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:43.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:43.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:43.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:43.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:43.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:43.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:44.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:44.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:44.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:44.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:44.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:44.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:44.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:44.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:45.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:45.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:45.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:45.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:45.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:45.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:45.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:45.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:46.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:46.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:46.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:46.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:46.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:46.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:46.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:46.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:47.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:47.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:47.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:47.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:47.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:47.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:47.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:47.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:48.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:48.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:48.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:48.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:48.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:48.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:48.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:48.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:49.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:49.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:49.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:49.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:49.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:49.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:49.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:49.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:50.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:50.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:50.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:50.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:50.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:50.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:50.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:50.889 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:51.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:51.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:51.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:51.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:51.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:51.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:51.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:51.880 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:52.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:52.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:52.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:52.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:52.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:52.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:52.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:52.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:53.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:53.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:53.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:53.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:53.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:53.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:53.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:53.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:54.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:54.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:54.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:54.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:54.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:54.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:54.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:54.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:55.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:55.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:55.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:55.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:55.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:55.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:55.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:55.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:56.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:56.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:56.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:56.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:56.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:56.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:56.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:56.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:57.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:57.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:57.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:57.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:57.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:57.533 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:57.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:57.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:58.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:58.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:58.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:58.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:58.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:58.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:58.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:58.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:59.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:59.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:59.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:59.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:59.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:59.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:59.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:59.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2020-09-08T16:53:59.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:53:59.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:00.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:00.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:00.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:00.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:00.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:00.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:00.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:00.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:01.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:01.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:01.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:01.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:01.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:01.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:02.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:02.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:02.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:02.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:02.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:02.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:02.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:02.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:03.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:03.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:03.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:03.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:03.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:03.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:03.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:03.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:04.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:04.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:04.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:04.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:04.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:04.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:04.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:04.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:05.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:05.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:05.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:05.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:05.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:05.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:05.872 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:05.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:06.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:06.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:06.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:06.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:06.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:06.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:06.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:06.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:07.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:07.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:07.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:07.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:07.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:07.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:07.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:07.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:08.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:08.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:08.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:08.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:08.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:08.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:08.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:08.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:09.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:09.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:09.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:09.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:09.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:09.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:09.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:09.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:10.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:10.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:10.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:10.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:10.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:10.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:10.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:10.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:11.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:11.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:11.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:11.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:11.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:11.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:11.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:11.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:12.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:12.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:12.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:12.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:12.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:12.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:12.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:12.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:13.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:13.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:13.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:13.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:13.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:13.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:13.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:13.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:14.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:14.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:14.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:14.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:14.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:14.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:14.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:15.042 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:15.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:15.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:15.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:15.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:15.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:15.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:15.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:15.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:15.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:16.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:16.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:16.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:16.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:16.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:16.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:16.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:16.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:17.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:17.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:17.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:17.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:17.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:17.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:17.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:17.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:18.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:18.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:18.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:18.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:18.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:18.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:18.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:18.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:19.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:19.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:19.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:19.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:19.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:19.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:19.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:19.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:20.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:20.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:20.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:20.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:20.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:20.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:20.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:20.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:21.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:21.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:21.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:21.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:21.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:21.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:21.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:21.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:22.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:22.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:22.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:22.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:22.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:22.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:22.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:22.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:23.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:23.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:23.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:23.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:23.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:23.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:24.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:24.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:24.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:24.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:24.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:24.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:24.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:24.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:25.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:25.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:25.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:25.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:25.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:25.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:25.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:25.903 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:26.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:26.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:26.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:26.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:26.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:26.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:26.994 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:26.994 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:27.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:27.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:27.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:27.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:27.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:27.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:28.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:28.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:28.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:28.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:28.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:28.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:28.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:28.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:29.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:29.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:29.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:29.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:29.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:29.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:29.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:29.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:30.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:30.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:30.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:30.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:30.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:30.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:31.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:31.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:31.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:31.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:31.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:31.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:31.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:31.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:32.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:32.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:32.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:32.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:32.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:32.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:33.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:33.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:33.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:33.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:34.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:34.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:34.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:34.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:34.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:34.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:34.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:34.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:35.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:35.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:35.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:35.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:35.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:35.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:35.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:36.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:36.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:36.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:36.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:36.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:36.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:36.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:37.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:37.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:37.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:37.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:37.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:37.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:38.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:38.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:38.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:38.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:38.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:38.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:38.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:38.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:39.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:39.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:39.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:39.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:39.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:39.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:40.039 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:40.040 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:40.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:537: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:54:40.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2020-09-08T16:54:40.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:542: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2020-09-08T16:54:40.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T16:54:40.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T16:54:40.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T16:54:40.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T16:54:40.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T16:54:40.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T16:54:40.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj250 2020-09-08T16:54:40.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T16:54:40.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T16:54:40.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T16:54:40.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2020-09-08T16:54:40.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T16:54:40.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:54:40.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj250 2020-09-08T16:54:40.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:54:40.632 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj250 2020-09-08T16:54:40.632 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj250 2020-09-08T16:54:40.632 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:54:41.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:54:41.007 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:54:41.007 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:54:41.007 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:54:41.008 INFO:tasks.workunit.client.0.smithi120.stderr:2147483647' 2020-09-08T16:54:41.008 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 4 0 2147483647 2020-09-08T16:54:41.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T16:54:41.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2020-09-08T16:54:41.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T16:54:41.009 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/3/type 2020-09-08T16:54:41.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T16:54:41.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2020-09-08T16:54:41.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T16:54:41.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2020-09-08T16:54:41.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T16:54:41.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T16:54:41.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T16:54:41.012 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2020-09-08T16:54:41.012 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T16:54:41.012 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T16:54:41.012 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:54:41.013 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:54:41.013 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:54:41.013 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T16:54:41.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2020-09-08T16:54:41.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T16:54:41.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T16:54:41.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T16:54:41.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2020-09-08T16:54:41.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T16:54:41.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T16:54:41.105 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:54:41.105 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:54:41.106 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:54:41.106 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T16:54:41.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T16:54:41.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2020-09-08T16:54:41.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:543: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2020-09-08T16:54:41.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T16:54:41.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T16:54:41.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T16:54:41.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T16:54:41.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T16:54:41.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T16:54:41.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj250 2020-09-08T16:54:41.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T16:54:41.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T16:54:41.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T16:54:41.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2020-09-08T16:54:41.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T16:54:41.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:54:41.206 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj250 2020-09-08T16:54:41.206 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:54:41.206 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj250 2020-09-08T16:54:41.206 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj250 2020-09-08T16:54:41.206 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:54:41.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:54:41.565 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:54:41.565 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:54:41.566 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:54:41.566 INFO:tasks.workunit.client.0.smithi120.stderr:2147483647' 2020-09-08T16:54:41.566 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 4 0 2147483647 2020-09-08T16:54:41.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T16:54:41.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2020-09-08T16:54:41.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T16:54:41.567 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/1/type 2020-09-08T16:54:41.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T16:54:41.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-09-08T16:54:41.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T16:54:41.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2020-09-08T16:54:41.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T16:54:41.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T16:54:41.570 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T16:54:41.570 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2020-09-08T16:54:41.570 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T16:54:41.570 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T16:54:41.570 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:54:41.571 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:54:41.571 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:54:41.571 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T16:54:41.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-09-08T16:54:41.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T16:54:41.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T16:54:41.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T16:54:41.664 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2020-09-08T16:54:41.664 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T16:54:41.665 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T16:54:41.665 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:54:41.665 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:54:41.665 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:54:41.665 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T16:54:41.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T16:54:41.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2020-09-08T16:54:41.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2020-09-08T16:54:41.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T16:54:41.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T16:54:41.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-09-08T16:54:41.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T16:54:41.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T16:54:41.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:54:41.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:54:41.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:54:41.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:54:41.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T16:54:41.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T16:54:41.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T16:54:41.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T16:54:41.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:54:41.766 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T16:54:41.766 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:54:41.766 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:54:41.766 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:54:41.767 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:54:41.767 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:54:41.767 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:54:41.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:54:41.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:54:41.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:54:41.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:54:41.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:54:41.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:54:41.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:54:41.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T16:54:41.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T16:54:41.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T16:54:41.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-09-08T16:54:41.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:54:41.770 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/2/whoami 2020-09-08T16:54:41.770 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T16:54:41.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-09-08T16:54:41.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T16:54:41.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T16:54:41.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T16:54:41.781 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:54:41.786+0000 7f37c4d55ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:54:41.782 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:54:41.788+0000 7f37c4d55ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:54:41.784 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:54:41.790+0000 7f37c4d55ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:54:42.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-09-08T16:54:42.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:54:42.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T16:54:42.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:54:42.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:54:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:54:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:54:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:54:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:54:42.130 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:54:42.300 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:54:42.306+0000 7f37c4d55ec0 -1 Falling back to public interface 2020-09-08T16:54:42.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:54:43.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:54:43.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:54:43.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:54:43.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:54:43.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:54:43.499 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:54:43.796 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:54:43.802+0000 7f37c4d55ec0 -1 osd.2 39 log_to_monitors {default=true} 2020-09-08T16:54:43.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:54:44.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:54:44.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:54:44.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:54:44.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:54:44.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:54:44.905 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:54:45.293 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up out weight 0 up_from 46 up_thru 20 down_at 40 last_clean_interval [18,39) [v2:127.0.0.1:6818/71255,v1:127.0.0.1:6819/71255] [v2:127.0.0.1:6820/71255,v1:127.0.0.1:6821/71255] exists,up 436ae6f4-f6e2-48c6-8f23-0ec18c0322ce 2020-09-08T16:54:45.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:54:45.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:54:45.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:54:45.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:546: TEST_ec_backfill_unfound: ceph osd in 2 2020-09-08T16:54:46.024 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 is already in. 2020-09-08T16:54:46.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: sleep 15 2020-09-08T16:55:01.035 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: seq 1 100 2020-09-08T16:55:01.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:01.036 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:01.036 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:01.037 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:01.037 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:01.038 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:01.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:01.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:01.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:01.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:01.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:01.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:01.401 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:02.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:02.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:02.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:02.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:02.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:02.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:02.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:02.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:02.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:02.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:02.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:02.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:02.766 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:03.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:03.767 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:03.767 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:03.767 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:03.767 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:03.768 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:04.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:04.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:04.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:04.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:04.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:04.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:04.134 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:05.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:05.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:05.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:05.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:05.145 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:05.145 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:05.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:05.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:05.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:05.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:05.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:05.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:05.500 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:06.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:06.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:06.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:06.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:06.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:06.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:06.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:06.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:06.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:06.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:06.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:06.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:06.867 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:07.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:07.868 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:07.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:07.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:07.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:07.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:08.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:08.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:08.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:08.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:08.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:08.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:08.234 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:09.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:09.237 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:09.237 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:09.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:09.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:09.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:09.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:09.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:09.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:09.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:09.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:09.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:09.603 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:10.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:10.604 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:10.604 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:10.604 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:10.604 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:10.605 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:10.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:10.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:10.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:10.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:10.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:10.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:10.966 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:11.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:11.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:11.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:11.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:11.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:11.969 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:12.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:12.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:12.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:12.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:12.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:12.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:12.334 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:13.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:13.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:13.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:13.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:13.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:13.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:13.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:13.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:13.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:13.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:13.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:13.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:13.723 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:14.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:14.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:14.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:14.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:14.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:14.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:15.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:15.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:15.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:15.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:15.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:15.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:15.089 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:16.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:16.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:16.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:16.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:16.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:16.091 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:16.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:16.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:16.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:16.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:16.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:16.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:16.456 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:17.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:17.457 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:17.457 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:17.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:17.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:17.459 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:17.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:17.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:17.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:17.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:17.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:17.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:17.826 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:18.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:18.827 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:18.827 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:18.827 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:18.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:18.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:19.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:19.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:19.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:19.197 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:20.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:20.198 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:20.198 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:20.198 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:20.199 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:20.199 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:20.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:20.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:20.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:20.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:20.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:20.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:20.559 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:21.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:21.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:21.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:21.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:21.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:21.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:21.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:21.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:21.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:21.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:21.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:21.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:21.939 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:22.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:22.933 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:22.934 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:22.934 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:22.934 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:22.934 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:23.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:23.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:23.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:23.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:23.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:23.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:23.300 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:24.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:24.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:24.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:24.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:24.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:24.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:24.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:24.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:24.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:24.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:24.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:24.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:24.666 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:25.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:25.667 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:25.667 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:25.668 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:25.668 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:25.668 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:26.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:26.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:26.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:26.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:26.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:26.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:26.033 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:27.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:27.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:27.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:27.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:27.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:27.029 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:27.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:27.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:27.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:27.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:27.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:27.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:27.393 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:28.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:28.394 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:28.394 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:28.394 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:28.394 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:28.395 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:28.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:28.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:28.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:28.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:28.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:28.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:28.755 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:29.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:29.756 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:29.756 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:29.757 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:29.757 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:29.757 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:30.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:30.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:30.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:30.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:30.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:30.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:30.124 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:31.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:31.121 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:31.122 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:31.122 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:31.122 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:31.123 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:31.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:31.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:31.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:31.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:31.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:31.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:31.494 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:32.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:32.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:32.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:32.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:32.495 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:32.496 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:32.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:32.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:32.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:32.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:32.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:32.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:32.863 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:33.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:33.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:33.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:33.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:33.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:33.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:34.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:34.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:34.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:34.227 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:34.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:34.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:34.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:35.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:35.229 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:35.229 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:35.229 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:35.230 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:35.230 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:35.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:35.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:35.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:35.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:35.592 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:36.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:36.592 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:36.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:36.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:36.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:36.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:36.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:36.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:36.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:36.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:36.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:36.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:36.960 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:37.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:37.961 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:37.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:37.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:37.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:37.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:38.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:38.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:38.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:38.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:38.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:38.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:38.328 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:39.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:39.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:39.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:39.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:39.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:39.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:39.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:39.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:39.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:39.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:39.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:39.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:39.694 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:40.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:40.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:40.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:40.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:40.696 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:40.696 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:41.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:41.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:41.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:41.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:41.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:41.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:41.060 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:42.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:42.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:42.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:42.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:42.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:42.062 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:42.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:42.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:42.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:42.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:42.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:42.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:42.430 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:43.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:43.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:43.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:43.432 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:43.432 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:43.432 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:43.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:43.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:43.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:43.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:43.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:43.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:43.798 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:44.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:44.800 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:44.800 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:44.800 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:44.800 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:44.801 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:45.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:45.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:45.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:45.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:45.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:45.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:45.166 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:46.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:46.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:46.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:46.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:46.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:46.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:46.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:46.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:46.524 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:46.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:46.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:46.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:47.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:47.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:47.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:47.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:47.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:47.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:47.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:47.889 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:47.889 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:47.890 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:47.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:47.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:47.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:48.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:48.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:48.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:48.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:48.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:48.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:49.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:49.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:49.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:49.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:49.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:49.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:49.260 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:50.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:50.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:50.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:50.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:50.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:50.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:50.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:50.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:50.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:50.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:50.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:50.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:50.630 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:51.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:51.636 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:51.636 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:51.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:51.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:51.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:51.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:51.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:51.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:51.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:51.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:51.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:51.999 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:52.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:52.999 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:53.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:53.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:53.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:53.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:53.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:53.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:53.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:53.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:53.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:53.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:53.362 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:54.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:54.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:54.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:54.364 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:54.364 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:54.364 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:54.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:54.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:54.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:54.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2020-09-08T16:55:54.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:54.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:557: TEST_ec_backfill_unfound: sleep 1 2020-09-08T16:55:54.729 INFO:tasks.workunit.client.0.smithi120.stdout:active+undersized+degraded+remapped+backfilling 2020-09-08T16:55:55.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: for tmp in $(seq 1 100) 2020-09-08T16:55:55.729 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: get_state 2.0 2020-09-08T16:55:55.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T16:55:55.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T16:55:55.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T16:55:55.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T16:55:56.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:551: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2020-09-08T16:55:56.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2020-09-08T16:55:56.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: grep backfill_unfound 2020-09-08T16:55:56.092 INFO:tasks.workunit.client.0.smithi120.stdout:active+backfill_unfound+undersized+degraded+remapped 2020-09-08T16:55:56.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2020-09-08T16:55:56.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: break 2020-09-08T16:55:56.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:560: TEST_ec_backfill_unfound: ceph pg dump pgs 2020-09-08T16:55:56.433 INFO:tasks.workunit.client.0.smithi120.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-09-08T16:55:56.433 INFO:tasks.workunit.client.0.smithi120.stdout:2.0 301 1 28 0 1 1232896 0 0 101 101 active+backfill_unfound+undersized+degraded+remapped 2020-09-08T16:55:51.764104+0000 45'301 53:897 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2020-09-08T16:53:13.398142+0000 0'0 2020-09-08T16:53:13.398142+0000 0 2020-09-08T16:55:56.433 INFO:tasks.workunit.client.0.smithi120.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:54:46.797784+0000 0'0 52:51 [4,1,2] 4 [4,1,2] 4 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:55:56.434 INFO:tasks.workunit.client.0.smithi120.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:13.742476+0000 0'0 52:75 [0,4,1] 0 [0,4,1] 0 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:55:56.434 INFO:tasks.workunit.client.0.smithi120.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:54:47.377942+0000 0'0 53:67 [1,0,2] 1 [1,0,2] 1 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:55:56.434 INFO:tasks.workunit.client.0.smithi120.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T16:53:12.870222+0000 0'0 52:27 [4,3,0] 4 [4,3,0] 4 0'0 2020-09-08T16:52:28.165882+0000 0'0 2020-09-08T16:52:28.165882+0000 0 2020-09-08T16:55:56.434 INFO:tasks.workunit.client.0.smithi120.stdout: 2020-09-08T16:55:56.434 INFO:tasks.workunit.client.0.smithi120.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-09-08T16:55:56.435 INFO:tasks.workunit.client.0.smithi120.stderr:dumped pgs 2020-09-08T16:55:56.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2020-09-08T16:55:56.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:561: TEST_ec_backfill_unfound: grep -q obj250 2020-09-08T16:55:56.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2020-09-08T16:56:01.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:565: TEST_ec_backfill_unfound: test 124 = 124 2020-09-08T16:56:01.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2020-09-08T16:56:01.865 INFO:tasks.workunit.client.0.smithi120.stderr:pg has no unfound objects 2020-09-08T16:56:01.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:569: TEST_ec_backfill_unfound: wait_for_clean 2020-09-08T16:56:01.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:56:01.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:56:01.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:56:01.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:56:01.874 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:56:01.875 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:56:01.875 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:56:01.875 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:56:01.876 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:56:01.876 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:56:01.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:56:01.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:56:01.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:56:01.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:56:01.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:56:02.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:56:02.329 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:56:02.329 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:56:02.329 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:56:02.329 INFO:tasks.workunit.client.0.smithi120.stderr:4' 2020-09-08T16:56:02.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:56:02.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:56:02.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:56:02.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803821 2020-09-08T16:56:02.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803821 2020-09-08T16:56:02.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803821' 2020-09-08T16:56:02.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:56:02.451 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:56:02.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607596 2020-09-08T16:56:02.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607596 2020-09-08T16:56:02.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803821 1-51539607596' 2020-09-08T16:56:02.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:56:02.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:56:02.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495634 2020-09-08T16:56:02.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495634 2020-09-08T16:56:02.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803821 1-51539607596 2-197568495634' 2020-09-08T16:56:02.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:56:02.693 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:56:02.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117032 2020-09-08T16:56:02.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117032 2020-09-08T16:56:02.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803821 1-51539607596 2-197568495634 3-115964117032' 2020-09-08T16:56:02.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:56:02.814 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:56:02.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=137438953511 2020-09-08T16:56:02.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 137438953511 2020-09-08T16:56:02.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803821 1-51539607596 2-197568495634 3-115964117032 4-137438953511' 2020-09-08T16:56:02.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:56:02.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803821 2020-09-08T16:56:02.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:56:02.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:56:02.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803821 2020-09-08T16:56:02.936 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:56:02.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803821 2020-09-08T16:56:02.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803821' 2020-09-08T16:56:02.937 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:56:02.937 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803821 2020-09-08T16:56:03.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803819 -lt 25769803821 2020-09-08T16:56:03.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:56:04.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T16:56:04.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:56:04.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803822 -lt 25769803821 2020-09-08T16:56:04.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:56:04.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607596 2020-09-08T16:56:04.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:56:04.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:56:04.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607596 2020-09-08T16:56:04.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:56:04.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607596 2020-09-08T16:56:04.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607596' 2020-09-08T16:56:04.728 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:56:04.728 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 51539607596 2020-09-08T16:56:05.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607596 -lt 51539607596 2020-09-08T16:56:05.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:56:05.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-197568495634 2020-09-08T16:56:05.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:56:05.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:56:05.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-197568495634 2020-09-08T16:56:05.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:56:05.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495634 2020-09-08T16:56:05.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 197568495634' 2020-09-08T16:56:05.289 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:56:05.290 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 197568495634 2020-09-08T16:56:05.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495634 -lt 197568495634 2020-09-08T16:56:05.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:56:05.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117032 2020-09-08T16:56:05.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:56:05.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:56:05.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117032 2020-09-08T16:56:05.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:56:05.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117032 2020-09-08T16:56:05.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117032' 2020-09-08T16:56:05.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:56:05.654 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 115964117032 2020-09-08T16:56:06.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117033 -lt 115964117032 2020-09-08T16:56:06.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:56:06.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-137438953511 2020-09-08T16:56:06.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:56:06.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:56:06.018 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-137438953511 2020-09-08T16:56:06.019 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:56:06.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=137438953511 2020-09-08T16:56:06.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 137438953511' 2020-09-08T16:56:06.019 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 137438953511 2020-09-08T16:56:06.020 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:56:06.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 137438953511 -lt 137438953511 2020-09-08T16:56:06.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:56:06.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:06.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:06.903 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T16:56:06.903 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:06.903 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:06.903 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:06.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:06.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:06.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:06.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:07.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:56:07.259 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:07.260 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:07.260 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:07.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:56:07.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' -1 2020-09-08T16:56:07.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T16:56:07.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=4 2020-09-08T16:56:07.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:56:07.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:56:07.884 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:56:07.884 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:07.884 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:07.884 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:07.885 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:07.885 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:07.885 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:07.885 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:08.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:56:08.240 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:08.241 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:08.241 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:08.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:56:08.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T16:56:08.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:56:08.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:56:08.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:56:08.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:56:08.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:56:08.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:56:08.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:56:09.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2047 2020-09-08T16:56:09.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2047 '!=' null 2020-09-08T16:56:09.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-09-08T16:56:09.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:56:09.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:56:09.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:56:09.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:09.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:09.328 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:09.328 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:09.328 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:09.328 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:09.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:09.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:56:09.687 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:09.687 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:09.687 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:10.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:56:10.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T16:56:10.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:56:10.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:56:10.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:56:10.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:56:10.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:56:10.228 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:56:10.229 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:56:10.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10239 2020-09-08T16:56:10.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10239 '!=' null 2020-09-08T16:56:10.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-09-08T16:56:10.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:56:10.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:56:10.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:56:10.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:10.807 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:10.807 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:10.807 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:10.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:10.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:10.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:11.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:56:11.161 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:11.161 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:11.161 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:11.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:56:11.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T16:56:11.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:56:11.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:56:11.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:56:11.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:56:11.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:56:11.639 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:56:11.639 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:56:12.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12287 2020-09-08T16:56:12.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12287 '!=' null 2020-09-08T16:56:12.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-09-08T16:56:12.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:56:12.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:56:12.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:56:12.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:12.232 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:12.232 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:12.232 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:12.233 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:12.233 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:12.233 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:12.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:56:12.584 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:12.584 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:12.584 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:13.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:56:13.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T16:56:13.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:56:13.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:56:13.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:56:13.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:56:13.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:56:13.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:56:13.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:56:13.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12287 2020-09-08T16:56:13.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12287 '!=' null 2020-09-08T16:56:13.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-09-08T16:56:13.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:56:13.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:56:13.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:56:13.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:13.654 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:13.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:13.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:13.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:13.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:13.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:14.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:56:14.024 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:14.024 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:14.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:14.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:56:14.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T16:56:14.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:56:14.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:56:14.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:56:14.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:56:14.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:56:14.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:56:14.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:56:15.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9728 2020-09-08T16:56:15.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9728 '!=' null 2020-09-08T16:56:15.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-09-08T16:56:15.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:56:15.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:56:15.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:56:15.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:15.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:15.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:15.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:15.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:15.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:15.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:15.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:56:15.460 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:15.460 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:15.460 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:15.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:56:15.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T16:56:15.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:56:15.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:56:15.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:56:15.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:56:15.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:56:15.943 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:56:15.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:56:16.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-09-08T16:56:16.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-09-08T16:56:16.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2020-09-08T16:56:16.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:56:16.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-09-08T16:56:16.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:56:16.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:56:16.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:56:16.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:56:16.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:56:16.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:56:16.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:56:16.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:56:16.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T16:56:16.978 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:56:16.979 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:56:16.979 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:56:17.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T16:56:17.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:56:17.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:56:17.456 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: seq 1 300 2020-09-08T16:56:17.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2020-09-08T16:56:17.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2020-09-08T16:56:17.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2020-09-08T16:56:17.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2020-09-08T16:56:17.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2020-09-08T16:56:17.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2020-09-08T16:56:17.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2020-09-08T16:56:17.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2020-09-08T16:56:17.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2020-09-08T16:56:17.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2020-09-08T16:56:17.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2020-09-08T16:56:17.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2020-09-08T16:56:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.802 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.802 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.803 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2020-09-08T16:56:17.803 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2020-09-08T16:56:17.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2020-09-08T16:56:17.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2020-09-08T16:56:17.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.900 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2020-09-08T16:56:17.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2020-09-08T16:56:17.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2020-09-08T16:56:17.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2020-09-08T16:56:17.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2020-09-08T16:56:17.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:17.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:17.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2020-09-08T16:56:18.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2020-09-08T16:56:18.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2020-09-08T16:56:18.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2020-09-08T16:56:18.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2020-09-08T16:56:18.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2020-09-08T16:56:18.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2020-09-08T16:56:18.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2020-09-08T16:56:18.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2020-09-08T16:56:18.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2020-09-08T16:56:18.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2020-09-08T16:56:18.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2020-09-08T16:56:18.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2020-09-08T16:56:18.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2020-09-08T16:56:18.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2020-09-08T16:56:18.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2020-09-08T16:56:18.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2020-09-08T16:56:18.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2020-09-08T16:56:18.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2020-09-08T16:56:18.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2020-09-08T16:56:18.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2020-09-08T16:56:18.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2020-09-08T16:56:18.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2020-09-08T16:56:18.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2020-09-08T16:56:18.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2020-09-08T16:56:18.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2020-09-08T16:56:18.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2020-09-08T16:56:18.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2020-09-08T16:56:18.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2020-09-08T16:56:18.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2020-09-08T16:56:18.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2020-09-08T16:56:18.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2020-09-08T16:56:18.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2020-09-08T16:56:18.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.880 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.880 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2020-09-08T16:56:18.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2020-09-08T16:56:18.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2020-09-08T16:56:18.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2020-09-08T16:56:18.983 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:18.984 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:18.984 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2020-09-08T16:56:18.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2020-09-08T16:56:19.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2020-09-08T16:56:19.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2020-09-08T16:56:19.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2020-09-08T16:56:19.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2020-09-08T16:56:19.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2020-09-08T16:56:19.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2020-09-08T16:56:19.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2020-09-08T16:56:19.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2020-09-08T16:56:19.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2020-09-08T16:56:19.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2020-09-08T16:56:19.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.277 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2020-09-08T16:56:19.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2020-09-08T16:56:19.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2020-09-08T16:56:19.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2020-09-08T16:56:19.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2020-09-08T16:56:19.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2020-09-08T16:56:19.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2020-09-08T16:56:19.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2020-09-08T16:56:19.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2020-09-08T16:56:19.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2020-09-08T16:56:19.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2020-09-08T16:56:19.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2020-09-08T16:56:19.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2020-09-08T16:56:19.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2020-09-08T16:56:19.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2020-09-08T16:56:19.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2020-09-08T16:56:19.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2020-09-08T16:56:19.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2020-09-08T16:56:19.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2020-09-08T16:56:19.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2020-09-08T16:56:19.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2020-09-08T16:56:19.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2020-09-08T16:56:19.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2020-09-08T16:56:19.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2020-09-08T16:56:19.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2020-09-08T16:56:19.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2020-09-08T16:56:19.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2020-09-08T16:56:19.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2020-09-08T16:56:19.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2020-09-08T16:56:19.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:19.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:19.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2020-09-08T16:56:19.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2020-09-08T16:56:20.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2020-09-08T16:56:20.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2020-09-08T16:56:20.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.079 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.079 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2020-09-08T16:56:20.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2020-09-08T16:56:20.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2020-09-08T16:56:20.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2020-09-08T16:56:20.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2020-09-08T16:56:20.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2020-09-08T16:56:20.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2020-09-08T16:56:20.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2020-09-08T16:56:20.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2020-09-08T16:56:20.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2020-09-08T16:56:20.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2020-09-08T16:56:20.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2020-09-08T16:56:20.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2020-09-08T16:56:20.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2020-09-08T16:56:20.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2020-09-08T16:56:20.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2020-09-08T16:56:20.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.482 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.484 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2020-09-08T16:56:20.484 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2020-09-08T16:56:20.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2020-09-08T16:56:20.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2020-09-08T16:56:20.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2020-09-08T16:56:20.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2020-09-08T16:56:20.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2020-09-08T16:56:20.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2020-09-08T16:56:20.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2020-09-08T16:56:20.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2020-09-08T16:56:20.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2020-09-08T16:56:20.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2020-09-08T16:56:20.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2020-09-08T16:56:20.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2020-09-08T16:56:20.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2020-09-08T16:56:20.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2020-09-08T16:56:20.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2020-09-08T16:56:20.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.899 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2020-09-08T16:56:20.900 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2020-09-08T16:56:20.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2020-09-08T16:56:20.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2020-09-08T16:56:20.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2020-09-08T16:56:20.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:20.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:20.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2020-09-08T16:56:20.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2020-09-08T16:56:21.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.046 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.046 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2020-09-08T16:56:21.046 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2020-09-08T16:56:21.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2020-09-08T16:56:21.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2020-09-08T16:56:21.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2020-09-08T16:56:21.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2020-09-08T16:56:21.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2020-09-08T16:56:21.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2020-09-08T16:56:21.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.242 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2020-09-08T16:56:21.242 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2020-09-08T16:56:21.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2020-09-08T16:56:21.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2020-09-08T16:56:21.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2020-09-08T16:56:21.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2020-09-08T16:56:21.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2020-09-08T16:56:21.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2020-09-08T16:56:21.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2020-09-08T16:56:21.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2020-09-08T16:56:21.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2020-09-08T16:56:21.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2020-09-08T16:56:21.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2020-09-08T16:56:21.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2020-09-08T16:56:21.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2020-09-08T16:56:21.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2020-09-08T16:56:21.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2020-09-08T16:56:21.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2020-09-08T16:56:21.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2020-09-08T16:56:21.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2020-09-08T16:56:21.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2020-09-08T16:56:21.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2020-09-08T16:56:21.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2020-09-08T16:56:21.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2020-09-08T16:56:21.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2020-09-08T16:56:21.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2020-09-08T16:56:21.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.897 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2020-09-08T16:56:21.898 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2020-09-08T16:56:21.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2020-09-08T16:56:21.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2020-09-08T16:56:21.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2020-09-08T16:56:21.994 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:21.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:21.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2020-09-08T16:56:21.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2020-09-08T16:56:22.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2020-09-08T16:56:22.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2020-09-08T16:56:22.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2020-09-08T16:56:22.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2020-09-08T16:56:22.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2020-09-08T16:56:22.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2020-09-08T16:56:22.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2020-09-08T16:56:22.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2020-09-08T16:56:22.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.261 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2020-09-08T16:56:22.261 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2020-09-08T16:56:22.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2020-09-08T16:56:22.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2020-09-08T16:56:22.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2020-09-08T16:56:22.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2020-09-08T16:56:22.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2020-09-08T16:56:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2020-09-08T16:56:22.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2020-09-08T16:56:22.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2020-09-08T16:56:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2020-09-08T16:56:22.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2020-09-08T16:56:22.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2020-09-08T16:56:22.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2020-09-08T16:56:22.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2020-09-08T16:56:22.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2020-09-08T16:56:22.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2020-09-08T16:56:22.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2020-09-08T16:56:22.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2020-09-08T16:56:22.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2020-09-08T16:56:22.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2020-09-08T16:56:22.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2020-09-08T16:56:22.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2020-09-08T16:56:22.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2020-09-08T16:56:22.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2020-09-08T16:56:22.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2020-09-08T16:56:22.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.897 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.897 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2020-09-08T16:56:22.897 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2020-09-08T16:56:22.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2020-09-08T16:56:22.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2020-09-08T16:56:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2020-09-08T16:56:22.994 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:22.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:22.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2020-09-08T16:56:22.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2020-09-08T16:56:23.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2020-09-08T16:56:23.044 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2020-09-08T16:56:23.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2020-09-08T16:56:23.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2020-09-08T16:56:23.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2020-09-08T16:56:23.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2020-09-08T16:56:23.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2020-09-08T16:56:23.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2020-09-08T16:56:23.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2020-09-08T16:56:23.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2020-09-08T16:56:23.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2020-09-08T16:56:23.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2020-09-08T16:56:23.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2020-09-08T16:56:23.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2020-09-08T16:56:23.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2020-09-08T16:56:23.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2020-09-08T16:56:23.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2020-09-08T16:56:23.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2020-09-08T16:56:23.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.484 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2020-09-08T16:56:23.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2020-09-08T16:56:23.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.533 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2020-09-08T16:56:23.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2020-09-08T16:56:23.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2020-09-08T16:56:23.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2020-09-08T16:56:23.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2020-09-08T16:56:23.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2020-09-08T16:56:23.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2020-09-08T16:56:23.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2020-09-08T16:56:23.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2020-09-08T16:56:23.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2020-09-08T16:56:23.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2020-09-08T16:56:23.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.815 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2020-09-08T16:56:23.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2020-09-08T16:56:23.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2020-09-08T16:56:23.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.880 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2020-09-08T16:56:23.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2020-09-08T16:56:23.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2020-09-08T16:56:23.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.953 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2020-09-08T16:56:23.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2020-09-08T16:56:23.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:23.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:23.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2020-09-08T16:56:23.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2020-09-08T16:56:24.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2020-09-08T16:56:24.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2020-09-08T16:56:24.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2020-09-08T16:56:24.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2020-09-08T16:56:24.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2020-09-08T16:56:24.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2020-09-08T16:56:24.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2020-09-08T16:56:24.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2020-09-08T16:56:24.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2020-09-08T16:56:24.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2020-09-08T16:56:24.270 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2020-09-08T16:56:24.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2020-09-08T16:56:24.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2020-09-08T16:56:24.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2020-09-08T16:56:24.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2020-09-08T16:56:24.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2020-09-08T16:56:24.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2020-09-08T16:56:24.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2020-09-08T16:56:24.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2020-09-08T16:56:24.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2020-09-08T16:56:24.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2020-09-08T16:56:24.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2020-09-08T16:56:24.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2020-09-08T16:56:24.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2020-09-08T16:56:24.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2020-09-08T16:56:24.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2020-09-08T16:56:24.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2020-09-08T16:56:24.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2020-09-08T16:56:24.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2020-09-08T16:56:24.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2020-09-08T16:56:24.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2020-09-08T16:56:24.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2020-09-08T16:56:24.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2020-09-08T16:56:24.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2020-09-08T16:56:24.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2020-09-08T16:56:24.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2020-09-08T16:56:24.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2020-09-08T16:56:24.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2020-09-08T16:56:24.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2020-09-08T16:56:24.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:24.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:24.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2020-09-08T16:56:24.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2020-09-08T16:56:25.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:25.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:25.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2020-09-08T16:56:25.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2020-09-08T16:56:25.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:25.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:25.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2020-09-08T16:56:25.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2020-09-08T16:56:25.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:25.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:571: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2020-09-08T16:56:25.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2020-09-08T16:56:25.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:577: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2020-09-08T16:56:25.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:25.079 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:582: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T16:56:25.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2020-09-08T16:56:25.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T16:56:25.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T16:56:25.545 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T16:56:25.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T16:56:26.088 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T16:56:26.096 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T16:56:26.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T16:56:26.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:56:26.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T16:56:26.097 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:56:26.098 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:56:26.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:56:26.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:56:26.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:56:26.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:56:26.207 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:56:26.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:56:26.208 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:56:26.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:56:26.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:56:26.209 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:56:26.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:56:26.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:56:26.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:56:26.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:56:26.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:56:26.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:56:26.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T16:56:26.242 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:56:26.242 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:26.242 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:26.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T16:56:26.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:56:26.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:56:26.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T16:56:26.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T16:56:26.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T16:56:26.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T16:56:26.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T16:56:26.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:56:26.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T16:56:26.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:56:26.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:56:26.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:56:26.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:56:26.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:56:26.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:56:26.249 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:56:26.249 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:56:26.249 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:56:26.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:56:26.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:56:26.250 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:56:26.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:56:26.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:56:26.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:56:26.251 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:56:26.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:56:26.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:56:26.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T16:56:26.253 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:56:26.253 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:26.253 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:26.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T16:56:26.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:56:26.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:56:26.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T16:56:26.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:56:26.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:26.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:26.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T16:56:26.257 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:56:26.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:56:26.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:56:26.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T16:56:26.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T16:56:26.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T16:56:26.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:56:26.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:56:26.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:56:26.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T16:56:26.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T16:56:26.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:56:26.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:56:26.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:56:26.620 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:56:26.621 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:26.622 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:26.623 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:26.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T16:56:26.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:56:26.789 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:56:26.789 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:56:26.789 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:56:26.789 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:56:26.790 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:56:26.790 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:56:26.790 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:56:26.790 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:56:26.791 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:56:26.791 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:26.791 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:26.791 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:56:26.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:56:26.792 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T16:56:26.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:56:26.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:56:26.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:56:26.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:56:26.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:56:26.879 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:56:26.879 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:56:26.879 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:56:26.879 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:56:26.879 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:26.880 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:26.880 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:56:26.880 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:56:26.881 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T16:56:26.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T16:56:26.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T16:56:26.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:56:26.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:56:26.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:56:26.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T16:56:26.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:56:27.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:56:27.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:56:27.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:56:27.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:56:27.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:27.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:27.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:27.848 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:56:27.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:56:27.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T16:56:27.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T16:56:28.156 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T16:56:28.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:56:29.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T16:56:29.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:56:29.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:56:29.167 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:56:29.167 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:29.167 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:29.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:56:29.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T16:56:29.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T16:56:29.247 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:56:29.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T16:56:29.256 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:56:26.653+0000 7fa99ceb66c0 0 load: jerasure load: lrc load: isa 2020-09-08T16:56:29.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2020-09-08T16:56:29.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:358: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2020-09-08T16:56:29.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local objname=myobject 2020-09-08T16:56:29.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:361: TEST_ec_object_attr_read_error: setup_osds 7 2020-09-08T16:56:29.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=7 2020-09-08T16:56:29.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T16:56:29.267 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 7 - 1 2020-09-08T16:56:29.267 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 6 2020-09-08T16:56:29.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:56:29.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T16:56:29.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:56:29.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:56:29.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:56:29.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:56:29.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T16:56:29.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:56:29.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:56:29.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:56:29.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:56:29.270 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T16:56:29.270 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T16:56:29.270 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:56:29.270 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:56:29.270 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:56:29.271 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:56:29.271 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:56:29.271 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:56:29.271 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:56:29.271 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:29.272 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:29.272 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:29.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:29.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:56:29.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:56:29.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:56:29.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:56:29.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:56:29.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:56:29.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:56:29.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:29.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:56:29.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:56:29.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T16:56:29.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:56:29.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a2fb4425-6979-4446-a87a-7d552d48e764 2020-09-08T16:56:29.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 a2fb4425-6979-4446-a87a-7d552d48e764' 2020-09-08T16:56:29.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:56:29.275 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 a2fb4425-6979-4446-a87a-7d552d48e764 2020-09-08T16:56:29.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC9t1dfCMS4EBAAV/+4xEeiaZm+HiHjkOMB8A== 2020-09-08T16:56:29.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC9t1dfCMS4EBAAV/+4xEeiaZm+HiHjkOMB8A=="}' 2020-09-08T16:56:29.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a2fb4425-6979-4446-a87a-7d552d48e764 -i td/test-erasure-eio/0/new.json 2020-09-08T16:56:29.639 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:56:29.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T16:56:29.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC9t1dfCMS4EBAAV/+4xEeiaZm+HiHjkOMB8A== --osd-uuid a2fb4425-6979-4446-a87a-7d552d48e764 2020-09-08T16:56:29.672 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:29.676+0000 7fb692341ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:29.673 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:29.678+0000 7fb692341ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:29.674 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:29.679+0000 7fb692341ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:29.676 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:29.680+0000 7fb692341ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:56:29.676 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:29.680+0000 7fb692341ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:56:29.676 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:29.680+0000 7fb692341ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:56:29.734 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:29.739+0000 7fb692341ec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T16:56:34.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T16:56:34.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:56:34.417 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:56:34.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:56:34.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:56:34.907 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:56:34.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:56:34.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:56:34.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:34.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:56:34.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:56:34.925 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:34.930+0000 7f412a0aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:34.927 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:34.932+0000 7f412a0aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:34.928 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:34.933+0000 7f412a0aeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:35.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:56:35.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:56:35.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:56:35.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:56:35.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:56:35.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:56:35.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:56:35.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:56:35.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:56:35.277 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:56:35.444 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:35.449+0000 7f412a0aeec0 -1 Falling back to public interface 2020-09-08T16:56:35.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:56:36.273 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:36.278+0000 7f412a0aeec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:56:36.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:56:36.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:56:36.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:56:36.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:56:36.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:56:36.677 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:56:37.053 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/81293,v1:127.0.0.1:6803/81293] [v2:127.0.0.1:6804/81293,v1:127.0.0.1:6805/81293] exists,up a2fb4425-6979-4446-a87a-7d552d48e764 2020-09-08T16:56:37.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:56:37.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:56:37.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:56:37.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:56:37.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T16:56:37.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:56:37.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:56:37.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T16:56:37.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:56:37.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T16:56:37.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:56:37.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:56:37.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:56:37.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:56:37.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T16:56:37.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T16:56:37.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:56:37.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:56:37.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:56:37.057 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:56:37.057 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:56:37.058 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:56:37.058 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:56:37.058 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:37.059 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:37.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:37.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:37.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:56:37.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:56:37.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:56:37.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:56:37.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:56:37.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:56:37.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:56:37.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:37.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:56:37.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:56:37.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T16:56:37.062 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:56:37.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=df015516-b4b3-49a4-bb8c-f1af3cc96a75 2020-09-08T16:56:37.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 df015516-b4b3-49a4-bb8c-f1af3cc96a75' 2020-09-08T16:56:37.063 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:56:37.064 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 df015516-b4b3-49a4-bb8c-f1af3cc96a75 2020-09-08T16:56:37.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDFt1dfcDVCBBAAWbQUaLzcyNFz3SGyOGfhnA== 2020-09-08T16:56:37.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDFt1dfcDVCBBAAWbQUaLzcyNFz3SGyOGfhnA=="}' 2020-09-08T16:56:37.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new df015516-b4b3-49a4-bb8c-f1af3cc96a75 -i td/test-erasure-eio/1/new.json 2020-09-08T16:56:37.809 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:56:37.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T16:56:37.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDFt1dfcDVCBBAAWbQUaLzcyNFz3SGyOGfhnA== --osd-uuid df015516-b4b3-49a4-bb8c-f1af3cc96a75 2020-09-08T16:56:37.840 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:37.844+0000 7f8be1a8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:37.859 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:37.863+0000 7f8be1a8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:37.860 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:37.864+0000 7f8be1a8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:37.860 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:37.864+0000 7f8be1a8dec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:56:37.860 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:37.864+0000 7f8be1a8dec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:56:37.860 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:37.864+0000 7f8be1a8dec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:56:37.920 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:37.926+0000 7f8be1a8dec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T16:56:42.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T16:56:42.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:56:42.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T16:56:42.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:56:42.653 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T16:56:43.143 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T16:56:43.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T16:56:43.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:56:43.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:43.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:56:43.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:56:43.162 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:43.168+0000 7f636dd2eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:43.164 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:43.169+0000 7f636dd2eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:43.164 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:43.170+0000 7f636dd2eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:43.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T16:56:43.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:56:43.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T16:56:43.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:56:43.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:56:43.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:56:43.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:56:43.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:56:43.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:56:43.512 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:56:43.680 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:43.686+0000 7f636dd2eec0 -1 Falling back to public interface 2020-09-08T16:56:43.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:56:44.467 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:44.473+0000 7f636dd2eec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T16:56:44.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:56:44.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:56:44.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:56:44.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:56:44.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:56:44.876 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:56:45.280 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/82290,v1:127.0.0.1:6811/82290] [v2:127.0.0.1:6812/82290,v1:127.0.0.1:6813/82290] exists,up df015516-b4b3-49a4-bb8c-f1af3cc96a75 2020-09-08T16:56:45.280 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 0ede037f-1953-40b8-a40d-e23a91f382ad 2020-09-08T16:56:45.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:56:45.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:56:45.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:56:45.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:56:45.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T16:56:45.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:56:45.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:56:45.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T16:56:45.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:56:45.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T16:56:45.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:56:45.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:56:45.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:56:45.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:56:45.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T16:56:45.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T16:56:45.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:56:45.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:56:45.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:56:45.284 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:56:45.284 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:56:45.285 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:56:45.285 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:56:45.285 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:45.285 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:45.285 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:45.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:45.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:56:45.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:56:45.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:56:45.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:56:45.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:56:45.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:56:45.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:56:45.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:45.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:56:45.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:56:45.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T16:56:45.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:56:45.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0ede037f-1953-40b8-a40d-e23a91f382ad 2020-09-08T16:56:45.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 0ede037f-1953-40b8-a40d-e23a91f382ad' 2020-09-08T16:56:45.289 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:56:45.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDNt1df3G+/ERAA7Z2NO5HWGsxrJoMnqT3mjg== 2020-09-08T16:56:45.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDNt1df3G+/ERAA7Z2NO5HWGsxrJoMnqT3mjg=="}' 2020-09-08T16:56:45.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0ede037f-1953-40b8-a40d-e23a91f382ad -i td/test-erasure-eio/2/new.json 2020-09-08T16:56:45.934 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:56:45.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T16:56:45.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDNt1df3G+/ERAA7Z2NO5HWGsxrJoMnqT3mjg== --osd-uuid 0ede037f-1953-40b8-a40d-e23a91f382ad 2020-09-08T16:56:45.965 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:45.971+0000 7f5df2ef2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:45.967 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:45.973+0000 7f5df2ef2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:45.969 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:45.974+0000 7f5df2ef2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:45.969 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:45.975+0000 7f5df2ef2ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:56:45.970 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:45.975+0000 7f5df2ef2ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:56:45.970 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:45.975+0000 7f5df2ef2ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:56:46.175 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:46.180+0000 7f5df2ef2ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T16:56:50.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T16:56:50.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:56:50.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T16:56:50.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:56:50.746 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T16:56:51.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T16:56:51.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:51.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:56:51.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:56:51.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:56:51.246 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T16:56:51.263 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:51.269+0000 7f4a24260ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:51.265 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:51.270+0000 7f4a24260ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:51.266 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:51.272+0000 7f4a24260ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:51.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T16:56:51.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:56:51.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T16:56:51.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:56:51.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:56:51.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:56:51.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:56:51.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:56:51.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:56:51.628 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:56:51.781 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:51.786+0000 7f4a24260ec0 -1 Falling back to public interface 2020-09-08T16:56:51.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:56:52.621 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:52.626+0000 7f4a24260ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T16:56:52.990 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:56:52.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:56:52.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:56:52.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:56:52.991 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:56:52.992 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:56:53.389 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/83284,v1:127.0.0.1:6819/83284] [v2:127.0.0.1:6820/83284,v1:127.0.0.1:6821/83284] exists,up 0ede037f-1953-40b8-a40d-e23a91f382ad 2020-09-08T16:56:53.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:56:53.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:56:53.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:56:53.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:56:53.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T16:56:53.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:56:53.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:56:53.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T16:56:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:56:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T16:56:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:56:53.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:56:53.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:56:53.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:56:53.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T16:56:53.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T16:56:53.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:56:53.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:56:53.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:56:53.394 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:56:53.395 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:56:53.395 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:56:53.395 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:56:53.395 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:56:53.395 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:56:53.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:53.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:56:53.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:56:53.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:56:53.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:56:53.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:56:53.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:56:53.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:56:53.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:56:53.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:53.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:56:53.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:56:53.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T16:56:53.398 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:56:53.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6e5f359e-199b-47cb-bad0-c3603c17e393 2020-09-08T16:56:53.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 6e5f359e-199b-47cb-bad0-c3603c17e393' 2020-09-08T16:56:53.400 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:56:53.400 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 6e5f359e-199b-47cb-bad0-c3603c17e393 2020-09-08T16:56:53.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDVt1dfz3pQGBAAC1/UmOox8uZkN2SKw3k1oQ== 2020-09-08T16:56:53.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDVt1dfz3pQGBAAC1/UmOox8uZkN2SKw3k1oQ=="}' 2020-09-08T16:56:53.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6e5f359e-199b-47cb-bad0-c3603c17e393 -i td/test-erasure-eio/3/new.json 2020-09-08T16:56:54.256 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T16:56:54.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T16:56:54.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDVt1dfz3pQGBAAC1/UmOox8uZkN2SKw3k1oQ== --osd-uuid 6e5f359e-199b-47cb-bad0-c3603c17e393 2020-09-08T16:56:54.289 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:54.294+0000 7f24c7f5bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:54.306 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:54.312+0000 7f24c7f5bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:54.308 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:54.313+0000 7f24c7f5bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:54.308 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:54.314+0000 7f24c7f5bec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:56:54.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:54.314+0000 7f24c7f5bec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:56:54.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:54.314+0000 7f24c7f5bec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:56:54.498 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:54.504+0000 7f24c7f5bec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T16:56:59.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T16:56:59.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:56:59.329 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T16:56:59.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T16:56:59.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:56:59.823 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T16:56:59.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T16:56:59.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:56:59.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:56:59.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:56:59.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:56:59.842 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:59.847+0000 7ff2e288cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:59.844 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:59.848+0000 7ff2e288cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:56:59.846 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:56:59.850+0000 7ff2e288cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:00.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T16:57:00.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:57:00.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T16:57:00.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:57:00.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:57:00.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:00.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:57:00.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:00.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:57:00.232 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:57:00.365 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:00.370+0000 7ff2e288cec0 -1 Falling back to public interface 2020-09-08T16:57:00.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:01.195 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:01.199+0000 7ff2e288cec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T16:57:01.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:01.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:01.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:57:01.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:01.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:57:01.595 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:57:01.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:02.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:02.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:02.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:57:02.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:02.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:57:02.968 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:57:03.401 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/84278,v1:127.0.0.1:6827/84278] [v2:127.0.0.1:6828/84278,v1:127.0.0.1:6829/84278] exists,up 6e5f359e-199b-47cb-bad0-c3603c17e393 2020-09-08T16:57:03.402 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 1a83eeb2-aee0-4620-9adb-c2eb7d8c42a2 2020-09-08T16:57:03.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:57:03.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:57:03.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:57:03.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:57:03.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 4 2020-09-08T16:57:03.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:57:03.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:57:03.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T16:57:03.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:57:03.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/4 2020-09-08T16:57:03.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:57:03.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:57:03.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:57:03.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:57:03.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2020-09-08T16:57:03.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2020-09-08T16:57:03.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:57:03.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:57:03.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:57:03.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:57:03.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:57:03.407 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:57:03.407 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:57:03.407 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:57:03.407 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:57:03.407 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:03.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:03.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:57:03.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:57:03.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:57:03.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:57:03.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:57:03.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:57:03.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:57:03.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:03.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:57:03.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:57:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/4 2020-09-08T16:57:03.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:57:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1a83eeb2-aee0-4620-9adb-c2eb7d8c42a2 2020-09-08T16:57:03.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 1a83eeb2-aee0-4620-9adb-c2eb7d8c42a2' 2020-09-08T16:57:03.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:57:03.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDft1dffD6tGBAAzFTMIwZ1ny6qhiJNEJH6Cg== 2020-09-08T16:57:03.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDft1dffD6tGBAAzFTMIwZ1ny6qhiJNEJH6Cg=="}' 2020-09-08T16:57:03.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1a83eeb2-aee0-4620-9adb-c2eb7d8c42a2 -i td/test-erasure-eio/4/new.json 2020-09-08T16:57:04.083 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T16:57:04.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/4/new.json 2020-09-08T16:57:04.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDft1dffD6tGBAAzFTMIwZ1ny6qhiJNEJH6Cg== --osd-uuid 1a83eeb2-aee0-4620-9adb-c2eb7d8c42a2 2020-09-08T16:57:04.115 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:04.120+0000 7f1bf9c1aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:04.117 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:04.122+0000 7f1bf9c1aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:04.118 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:04.123+0000 7f1bf9c1aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:04.119 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:04.124+0000 7f1bf9c1aec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:57:04.119 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:04.124+0000 7f1bf9c1aec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:57:04.120 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:04.124+0000 7f1bf9c1aec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:57:04.313 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:04.317+0000 7f1bf9c1aec0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2020-09-08T16:57:08.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2020-09-08T16:57:08.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:57:08.941 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T16:57:08.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T16:57:08.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:57:09.436 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T16:57:09.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T16:57:09.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:09.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:57:09.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:57:09.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:57:09.455 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:09.459+0000 7f8585169ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:09.456 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:09.461+0000 7f8585169ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:09.457 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:09.462+0000 7f8585169ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:09.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T16:57:09.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:57:09.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T16:57:09.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:57:09.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:57:09.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:09.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:57:09.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:09.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:57:09.818 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:57:09.977 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:09.981+0000 7f8585169ec0 -1 Falling back to public interface 2020-09-08T16:57:10.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:10.769 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:10.774+0000 7f8585169ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T16:57:11.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:11.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:11.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:57:11.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:11.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:57:11.182 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:57:11.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:12.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:12.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:12.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:57:12.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:12.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:57:12.567 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:57:13.051 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 33 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/85298,v1:127.0.0.1:6835/85298] [v2:127.0.0.1:6836/85298,v1:127.0.0.1:6837/85298] exists,up 1a83eeb2-aee0-4620-9adb-c2eb7d8c42a2 2020-09-08T16:57:13.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:57:13.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:57:13.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:57:13.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:57:13.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 5 2020-09-08T16:57:13.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:57:13.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:57:13.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-09-08T16:57:13.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:57:13.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/5 2020-09-08T16:57:13.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:57:13.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:57:13.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:57:13.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:57:13.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2020-09-08T16:57:13.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2020-09-08T16:57:13.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:57:13.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:57:13.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:57:13.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:57:13.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:57:13.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:57:13.056 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:57:13.056 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:57:13.056 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:57:13.056 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:13.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:13.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:57:13.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:57:13.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:57:13.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:57:13.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:57:13.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:57:13.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:57:13.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:13.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:57:13.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:57:13.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/5 2020-09-08T16:57:13.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:57:13.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7b07e9b3-eb3d-4dcc-b3ef-fb4044b5b7f9 2020-09-08T16:57:13.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 7b07e9b3-eb3d-4dcc-b3ef-fb4044b5b7f9' 2020-09-08T16:57:13.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:57:13.060 INFO:tasks.workunit.client.0.smithi120.stdout:add osd5 7b07e9b3-eb3d-4dcc-b3ef-fb4044b5b7f9 2020-09-08T16:57:13.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDpt1dfj1grBBAAbOqMf8ptxD5bsZAko260KA== 2020-09-08T16:57:13.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDpt1dfj1grBBAAbOqMf8ptxD5bsZAko260KA=="}' 2020-09-08T16:57:13.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7b07e9b3-eb3d-4dcc-b3ef-fb4044b5b7f9 -i td/test-erasure-eio/5/new.json 2020-09-08T16:57:13.653 INFO:tasks.workunit.client.0.smithi120.stdout:5 2020-09-08T16:57:13.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/5/new.json 2020-09-08T16:57:13.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpt1dfj1grBBAAbOqMf8ptxD5bsZAko260KA== --osd-uuid 7b07e9b3-eb3d-4dcc-b3ef-fb4044b5b7f9 2020-09-08T16:57:13.687 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:13.689+0000 7f12a7cb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:13.689 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:13.694+0000 7f12a7cb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:13.690 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:13.694+0000 7f12a7cb7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:13.690 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:13.695+0000 7f12a7cb7ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T16:57:13.691 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:13.695+0000 7f12a7cb7ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T16:57:13.691 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:13.695+0000 7f12a7cb7ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T16:57:13.825 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:13.830+0000 7f12a7cb7ec0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2020-09-08T16:57:18.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2020-09-08T16:57:18.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:57:18.740 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd5 key to auth repository 2020-09-08T16:57:18.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-09-08T16:57:18.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:57:19.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-09-08T16:57:19.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:57:19.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:19.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:57:19.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:57:19.239 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.5 2020-09-08T16:57:19.256 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:19.261+0000 7efe6eb10ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:19.258 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:19.263+0000 7efe6eb10ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:19.260 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:19.264+0000 7efe6eb10ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:19.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-09-08T16:57:19.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:57:19.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-09-08T16:57:19.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:57:19.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:57:19.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:19.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:57:19.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:19.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:57:19.624 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:57:19.779 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:19.784+0000 7efe6eb10ec0 -1 Falling back to public interface 2020-09-08T16:57:20.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:20.631 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:20.636+0000 7efe6eb10ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-09-08T16:57:21.042 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:21.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:21.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:57:21.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:21.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:57:21.044 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:57:21.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:22.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:22.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:22.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:57:22.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:22.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:57:22.526 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:57:22.967 INFO:tasks.workunit.client.0.smithi120.stdout:osd.5 up in weight 1 up_from 40 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/86318,v1:127.0.0.1:6843/86318] [v2:127.0.0.1:6844/86318,v1:127.0.0.1:6845/86318] exists,up 7b07e9b3-eb3d-4dcc-b3ef-fb4044b5b7f9 2020-09-08T16:57:22.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:57:22.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:57:22.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:57:22.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:57:22.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 6 2020-09-08T16:57:22.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:57:22.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:57:22.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2020-09-08T16:57:22.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:57:22.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/6 2020-09-08T16:57:22.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:57:22.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:57:22.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:57:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:57:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2020-09-08T16:57:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2020-09-08T16:57:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:57:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:57:22.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:57:22.972 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:57:22.972 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:57:22.972 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:57:22.972 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:57:22.972 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:57:22.973 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:57:22.973 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:22.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:22.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:57:22.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:57:22.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:57:22.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:57:22.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:57:22.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:57:22.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:57:22.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:22.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:57:22.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:57:22.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/6 2020-09-08T16:57:22.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:57:22.977 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3d4e525a-c935-4922-b827-1c972862939b 2020-09-08T16:57:22.977 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 3d4e525a-c935-4922-b827-1c972862939b' 2020-09-08T16:57:22.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:57:22.978 INFO:tasks.workunit.client.0.smithi120.stdout:add osd6 3d4e525a-c935-4922-b827-1c972862939b 2020-09-08T16:57:22.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDyt1dfyoHNOhAAcBeU7MbQBpcIOZT8iBEUzQ== 2020-09-08T16:57:22.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDyt1dfyoHNOhAAcBeU7MbQBpcIOZT8iBEUzQ=="}' 2020-09-08T16:57:22.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3d4e525a-c935-4922-b827-1c972862939b -i td/test-erasure-eio/6/new.json 2020-09-08T16:57:23.514 INFO:tasks.workunit.client.0.smithi120.stdout:6 2020-09-08T16:57:23.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/6/new.json 2020-09-08T16:57:23.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDyt1dfyoHNOhAAcBeU7MbQBpcIOZT8iBEUzQ== --osd-uuid 3d4e525a-c935-4922-b827-1c972862939b 2020-09-08T16:57:23.547 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:23.551+0000 7fcdf7a0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:23.604 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:23.608+0000 7fcdf7a0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:23.605 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:23.610+0000 7fcdf7a0aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:23.606 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:23.611+0000 7fcdf7a0aec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T16:57:23.606 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:23.611+0000 7fcdf7a0aec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T16:57:23.607 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:23.611+0000 7fcdf7a0aec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T16:57:23.786 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:23.792+0000 7fcdf7a0aec0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2020-09-08T16:57:28.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2020-09-08T16:57:28.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:57:28.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2020-09-08T16:57:28.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:57:28.732 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd6 key to auth repository 2020-09-08T16:57:29.226 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.6 2020-09-08T16:57:29.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2020-09-08T16:57:29.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:29.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:57:29.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:57:29.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:57:29.245 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:29.251+0000 7f880ff2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:29.247 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:29.252+0000 7f880ff2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:29.248 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:29.253+0000 7f880ff2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:29.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2020-09-08T16:57:29.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:57:29.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-09-08T16:57:29.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:57:29.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:57:29.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:29.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:57:29.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:29.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:57:29.610 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:57:29.773 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:29.779+0000 7f880ff2fec0 -1 Falling back to public interface 2020-09-08T16:57:29.978 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:30.576 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:30.581+0000 7f880ff2fec0 -1 osd.6 0 log_to_monitors {default=true} 2020-09-08T16:57:30.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:30.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:30.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:57:30.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:30.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:57:30.981 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:57:31.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:32.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:32.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:32.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:57:32.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:32.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:57:32.361 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:57:32.817 INFO:tasks.workunit.client.0.smithi120.stdout:osd.6 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/87343,v1:127.0.0.1:6851/87343] [v2:127.0.0.1:6852/87343,v1:127.0.0.1:6853/87343] exists,up 3d4e525a-c935-4922-b827-1c972862939b 2020-09-08T16:57:32.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:57:32.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:57:32.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:57:32.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T16:57:32.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:57:32.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:57:32.818 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:57:32.819 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:57:32.819 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:57:32.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T16:57:32.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T16:57:32.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T16:57:32.907 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:57:32.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T16:57:32.917 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:56:35.455+0000 7f412a0aeec0 0 load: jerasure load: lrc load: isa 2020-09-08T16:57:32.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:363: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2020-09-08T16:57:32.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2020-09-08T16:57:32.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T16:57:32.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T16:57:32.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=3 2020-09-08T16:57:32.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T16:57:32.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=2 2020-09-08T16:57:32.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T16:57:32.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2020-09-08T16:57:33.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T16:57:33.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T16:57:34.353 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T16:57:34.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:57:35.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T16:57:35.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:57:35.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:57:35.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:57:35.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:57:35.364 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:57:35.365 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:57:35.365 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:57:35.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:57:35.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:57:35.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:57:35.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:57:35.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:57:35.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:57:35.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:57:35.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:57:35.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:57:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:57:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:57:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:57:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:57:35.828 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:57:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T16:57:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:57:35.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:35.829 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:57:35.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803791 2020-09-08T16:57:35.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803791 2020-09-08T16:57:35.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791' 2020-09-08T16:57:35.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:35.950 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:57:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574862 2020-09-08T16:57:36.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574862 2020-09-08T16:57:36.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-55834574862' 2020-09-08T16:57:36.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:36.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:57:36.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411339 2020-09-08T16:57:36.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411339 2020-09-08T16:57:36.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-55834574862 2-77309411339' 2020-09-08T16:57:36.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:36.269 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:57:36.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149705 2020-09-08T16:57:36.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149705 2020-09-08T16:57:36.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-55834574862 2-77309411339 3-111669149705' 2020-09-08T16:57:36.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:36.390 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:57:36.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920775 2020-09-08T16:57:36.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920775 2020-09-08T16:57:36.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-55834574862 2-77309411339 3-111669149705 4-141733920775' 2020-09-08T16:57:36.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:36.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:57:36.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691845 2020-09-08T16:57:36.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691845 2020-09-08T16:57:36.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-55834574862 2-77309411339 3-111669149705 4-141733920775 5-171798691845' 2020-09-08T16:57:36.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:36.632 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:57:36.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495619 2020-09-08T16:57:36.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495619 2020-09-08T16:57:36.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-55834574862 2-77309411339 3-111669149705 4-141733920775 5-171798691845 6-197568495619' 2020-09-08T16:57:36.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:36.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803791 2020-09-08T16:57:36.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:36.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:57:36.754 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803791 2020-09-08T16:57:36.754 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:36.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803791 2020-09-08T16:57:36.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803791' 2020-09-08T16:57:36.755 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:57:36.755 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803791 2020-09-08T16:57:37.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803789 -lt 25769803791 2020-09-08T16:57:37.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:57:38.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T16:57:38.116 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:57:38.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803791 -lt 25769803791 2020-09-08T16:57:38.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:38.488 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574862 2020-09-08T16:57:38.488 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:38.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:57:38.489 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574862 2020-09-08T16:57:38.489 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:38.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574862 2020-09-08T16:57:38.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574862' 2020-09-08T16:57:38.490 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:57:38.491 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574862 2020-09-08T16:57:38.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574862 -lt 55834574862 2020-09-08T16:57:38.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:38.853 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411339 2020-09-08T16:57:38.854 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:38.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:57:38.855 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411339 2020-09-08T16:57:38.855 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:38.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411339 2020-09-08T16:57:38.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411339' 2020-09-08T16:57:38.856 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:57:38.856 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411339 2020-09-08T16:57:39.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411339 -lt 77309411339 2020-09-08T16:57:39.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:39.234 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149705 2020-09-08T16:57:39.235 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:39.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:57:39.235 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149705 2020-09-08T16:57:39.236 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:39.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149705 2020-09-08T16:57:39.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149705' 2020-09-08T16:57:39.237 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:57:39.237 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149705 2020-09-08T16:57:39.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149706 -lt 111669149705 2020-09-08T16:57:39.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:39.634 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920775 2020-09-08T16:57:39.635 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:39.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:57:39.635 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920775 2020-09-08T16:57:39.636 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:39.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920775 2020-09-08T16:57:39.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920775' 2020-09-08T16:57:39.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:57:39.637 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920775 2020-09-08T16:57:40.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920776 -lt 141733920775 2020-09-08T16:57:40.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:40.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-171798691845 2020-09-08T16:57:40.028 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:40.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:57:40.029 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-171798691845 2020-09-08T16:57:40.029 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:40.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691845 2020-09-08T16:57:40.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 171798691845' 2020-09-08T16:57:40.030 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:57:40.031 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 171798691845 2020-09-08T16:57:40.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691846 -lt 171798691845 2020-09-08T16:57:40.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:40.404 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-197568495619 2020-09-08T16:57:40.404 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:40.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:57:40.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-197568495619 2020-09-08T16:57:40.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:40.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495619 2020-09-08T16:57:40.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 197568495619' 2020-09-08T16:57:40.406 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 197568495619 2020-09-08T16:57:40.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:57:40.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495620 -lt 197568495619 2020-09-08T16:57:40.767 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:57:40.767 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:57:40.768 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:57:41.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T16:57:41.253 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:57:41.254 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:57:41.254 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:57:41.254 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:57:41.254 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:57:41.254 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:57:41.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:57:41.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T16:57:41.609 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:57:41.609 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:57:41.610 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:57:42.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T16:57:42.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:57:42.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:57:42.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:366: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2020-09-08T16:57:42.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=pool-jerasure 2020-09-08T16:57:42.106 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=myobject 2020-09-08T16:57:42.106 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map pool-jerasure myobject 2020-09-08T16:57:42.106 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2020-09-08T16:57:42.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:366: TEST_ec_object_attr_read_error: local primary_osd=3 2020-09-08T16:57:42.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:368: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2020-09-08T16:57:42.480 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:57:42.481 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:57:42.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:57:42.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:57:42.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:57:42.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:57:42.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:371: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2020-09-08T16:57:42.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T16:57:42.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T16:57:42.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=myobject 2020-09-08T16:57:42.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:57:42.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T16:57:42.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:57:42.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T16:57:42.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:57:42.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T16:57:42.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:57:42.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T16:57:42.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2020-09-08T16:57:52.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:374: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2020-09-08T16:57:52.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T16:57:52.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T16:57:52.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=mdata 2020-09-08T16:57:52.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T16:57:52.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T16:57:52.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T16:57:52.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=myobject 2020-09-08T16:57:52.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T16:57:52.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T16:57:52.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T16:57:52.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2020-09-08T16:57:52.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T16:57:52.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:57:52.762 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure myobject 2020-09-08T16:57:52.762 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:57:52.762 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T16:57:52.762 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T16:57:52.762 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:57:53.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2147483647 2020-09-08T16:57:53.148 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:57:53.148 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:57:53.148 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:57:53.149 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T16:57:53.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2147483647 5 1 0 6 2020-09-08T16:57:53.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T16:57:53.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=5 2020-09-08T16:57:53.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T16:57:53.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/5/type 2020-09-08T16:57:53.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T16:57:53.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2020-09-08T16:57:53.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T16:57:53.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=5 2020-09-08T16:57:53.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T16:57:53.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T16:57:53.153 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T16:57:53.153 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.5 2020-09-08T16:57:53.153 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2020-09-08T16:57:53.153 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2020-09-08T16:57:53.153 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:57:53.154 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:57:53.154 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:57:53.154 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.5.asok 2020-09-08T16:57:53.154 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2020-09-08T16:57:53.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T16:57:53.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T16:57:53.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T16:57:53.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.5 2020-09-08T16:57:53.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2020-09-08T16:57:53.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2020-09-08T16:57:53.247 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:57:53.248 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:57:53.248 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:57:53.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.5.asok 2020-09-08T16:57:53.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T16:57:53.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2020-09-08T16:57:53.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:377: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2020-09-08T16:57:53.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T16:57:53.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T16:57:53.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-09-08T16:57:53.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T16:57:53.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T16:57:53.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:57:53.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:57:53.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:57:53.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:57:53.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T16:57:53.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T16:57:53.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T16:57:53.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T16:57:53.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:57:53.349 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T16:57:53.349 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:57:53.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:57:53.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:57:53.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:57:53.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:57:53.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:53.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:57:53.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:57:53.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:57:53.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:57:53.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:57:53.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:57:53.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:53.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T16:57:53.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T16:57:53.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T16:57:53.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-09-08T16:57:53.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:57:53.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/3/whoami 2020-09-08T16:57:53.353 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T16:57:53.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-09-08T16:57:53.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T16:57:53.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T16:57:53.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T16:57:53.364 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:53.369+0000 7fcda181eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:53.366 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:53.371+0000 7fcda181eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:53.368 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:53.373+0000 7fcda181eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:57:53.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-09-08T16:57:53.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:57:53.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T16:57:53.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:57:53.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:57:53.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:53.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:57:53.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:53.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:57:53.717 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:57:53.889 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:53.894+0000 7fcda181eec0 -1 Falling back to public interface 2020-09-08T16:57:54.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:57:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:57:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:57:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:57:55.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:57:55.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:57:55.119 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:57:55.120 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:57:55.124+0000 7fcda181eec0 -1 osd.3 54 log_to_monitors {default=true} 2020-09-08T16:57:55.557 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 57 up_thru 52 down_at 55 last_clean_interval [26,54) [v2:127.0.0.1:6826/88536,v1:127.0.0.1:6827/88536] [v2:127.0.0.1:6828/88536,v1:127.0.0.1:6829/88536] exists,up 6e5f359e-199b-47cb-bad0-c3603c17e393 2020-09-08T16:57:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:57:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:57:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:57:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:380: TEST_ec_object_attr_read_error: wait_for_clean 2020-09-08T16:57:55.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:57:55.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:57:55.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:57:55.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:57:55.559 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:57:55.560 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:57:55.560 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:57:55.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:57:55.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:57:55.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:57:55.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:57:55.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:57:55.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:57:55.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:57:55.649 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:57:56.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:57:56.102 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:57:56.102 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:57:56.102 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:57:56.103 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:57:56.103 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:57:56.103 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T16:57:56.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:57:56.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:56.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:57:56.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803797 2020-09-08T16:57:56.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803797 2020-09-08T16:57:56.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797' 2020-09-08T16:57:56.223 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:56.224 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:57:56.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574868 2020-09-08T16:57:56.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574868 2020-09-08T16:57:56.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-55834574868' 2020-09-08T16:57:56.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:56.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:57:56.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411345 2020-09-08T16:57:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411345 2020-09-08T16:57:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-55834574868 2-77309411345' 2020-09-08T16:57:56.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:56.466 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:57:56.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=244813135875 2020-09-08T16:57:56.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 244813135875 2020-09-08T16:57:56.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-55834574868 2-77309411345 3-244813135875' 2020-09-08T16:57:56.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:56.585 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:57:56.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920781 2020-09-08T16:57:56.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920781 2020-09-08T16:57:56.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-55834574868 2-77309411345 3-244813135875 4-141733920781' 2020-09-08T16:57:56.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:56.706 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:57:56.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691851 2020-09-08T16:57:56.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691851 2020-09-08T16:57:56.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-55834574868 2-77309411345 3-244813135875 4-141733920781 5-171798691851' 2020-09-08T16:57:56.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:57:56.835 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:57:56.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495626 2020-09-08T16:57:56.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495626 2020-09-08T16:57:56.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-55834574868 2-77309411345 3-244813135875 4-141733920781 5-171798691851 6-197568495626' 2020-09-08T16:57:56.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:56.946 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803797 2020-09-08T16:57:56.946 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:56.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:57:56.946 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803797 2020-09-08T16:57:56.947 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:56.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803797 2020-09-08T16:57:56.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2020-09-08T16:57:56.948 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:57:56.948 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803797 2020-09-08T16:57:57.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803795 -lt 25769803797 2020-09-08T16:57:57.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T16:57:58.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T16:57:58.317 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:57:58.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803797 -lt 25769803797 2020-09-08T16:57:58.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:58.693 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574868 2020-09-08T16:57:58.693 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:58.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:57:58.694 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574868 2020-09-08T16:57:58.694 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:58.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574868 2020-09-08T16:57:58.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2020-09-08T16:57:58.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:57:58.695 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574868 2020-09-08T16:57:59.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574868 -lt 55834574868 2020-09-08T16:57:59.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:59.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411345 2020-09-08T16:57:59.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:59.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:57:59.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411345 2020-09-08T16:57:59.062 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:59.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411345 2020-09-08T16:57:59.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411345' 2020-09-08T16:57:59.063 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:57:59.063 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411345 2020-09-08T16:57:59.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411345 -lt 77309411345 2020-09-08T16:57:59.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:59.463 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-244813135875 2020-09-08T16:57:59.463 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:59.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:57:59.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-244813135875 2020-09-08T16:57:59.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:59.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=244813135875 2020-09-08T16:57:59.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 244813135875' 2020-09-08T16:57:59.466 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:57:59.466 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 244813135875 2020-09-08T16:57:59.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 244813135875 -lt 244813135875 2020-09-08T16:57:59.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:57:59.830 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920781 2020-09-08T16:57:59.831 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:57:59.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:57:59.831 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920781 2020-09-08T16:57:59.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:57:59.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920781 2020-09-08T16:57:59.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920781' 2020-09-08T16:57:59.833 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:57:59.833 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920781 2020-09-08T16:58:00.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920782 -lt 141733920781 2020-09-08T16:58:00.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:58:00.207 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-171798691851 2020-09-08T16:58:00.207 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:58:00.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:58:00.208 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-171798691851 2020-09-08T16:58:00.208 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:58:00.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691851 2020-09-08T16:58:00.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 171798691851' 2020-09-08T16:58:00.210 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:58:00.210 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 171798691851 2020-09-08T16:58:00.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691852 -lt 171798691851 2020-09-08T16:58:00.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:58:00.577 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-197568495626 2020-09-08T16:58:00.577 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:58:00.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:58:00.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-197568495626 2020-09-08T16:58:00.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:58:00.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495626 2020-09-08T16:58:00.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 197568495626' 2020-09-08T16:58:00.579 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:58:00.579 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 197568495626 2020-09-08T16:58:00.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495626 -lt 197568495626 2020-09-08T16:58:00.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:58:00.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:58:00.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:58:01.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T16:58:01.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:58:01.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:58:01.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:58:01.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:58:01.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:58:01.427 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:58:01.427 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:58:01.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T16:58:01.781 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:58:01.781 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:58:01.782 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:58:02.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T16:58:02.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:58:02.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:58:02.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:382: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2020-09-08T16:58:02.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T16:58:02.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T16:58:02.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=myobject 2020-09-08T16:58:02.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T16:58:02.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T16:58:02.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2020-09-08T16:58:07.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T16:58:07.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T16:58:07.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:384: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2020-09-08T16:58:07.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T16:58:07.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T16:58:08.462 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T16:58:08.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T16:58:09.044 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T16:58:09.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T16:58:09.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T16:58:09.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:58:09.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T16:58:09.049 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:58:09.049 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:58:09.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:58:09.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:58:09.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:58:09.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:58:09.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:58:09.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:58:09.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:58:09.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:58:09.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:58:09.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:58:09.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:58:09.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:58:09.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:58:09.170 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:58:09.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:58:09.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:58:09.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T16:58:09.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:58:09.180 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:09.180 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:09.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T16:58:09.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:58:09.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:58:09.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T16:58:09.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T16:58:09.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T16:58:09.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T16:58:09.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T16:58:09.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:58:09.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T16:58:09.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:58:09.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:58:09.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:58:09.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:58:09.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:58:09.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:58:09.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:58:09.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:58:09.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:58:09.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:58:09.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:58:09.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:58:09.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:58:09.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:58:09.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:58:09.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:58:09.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:58:09.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:58:09.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T16:58:09.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:58:09.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:09.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:09.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T16:58:09.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:58:09.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:58:09.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T16:58:09.194 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:58:09.194 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:09.194 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:09.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T16:58:09.195 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:58:09.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:58:09.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:58:09.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T16:58:09.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T16:58:09.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T16:58:09.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:58:09.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:58:09.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:58:09.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T16:58:09.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T16:58:09.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:58:09.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:09.500 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:09.500 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:09.500 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:09.500 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:09.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:09.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T16:58:09.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:58:09.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:58:09.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:58:09.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:58:09.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:58:09.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:58:09.720 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:58:09.720 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:58:09.720 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:58:09.720 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:58:09.720 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:09.721 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:09.721 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:58:09.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:58:09.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T16:58:09.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:58:09.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:58:09.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:58:09.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:58:09.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:58:09.810 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:58:09.810 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:58:09.810 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:58:09.810 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:58:09.811 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:09.811 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:09.811 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:58:09.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:58:09.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T16:58:09.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T16:58:09.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T16:58:09.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:58:09.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:58:09.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:58:09.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T16:58:09.896 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T16:58:10.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T16:58:10.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:10.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:10.819 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:10.819 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:10.820 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:10.820 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:10.820 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:58:10.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T16:58:10.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T16:58:10.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T16:58:11.136 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T16:58:11.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:58:12.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T16:58:12.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:58:12.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:58:12.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:58:12.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:12.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:12.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:58:12.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T16:58:12.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T16:58:12.228 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:58:12.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T16:58:12.238 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:58:09.534+0000 7fb92ec206c0 0 load: jerasure load: lrc load: isa 2020-09-08T16:58:12.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2020-09-08T16:58:12.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:417: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2020-09-08T16:58:12.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local objname=myobject 2020-09-08T16:58:12.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:420: TEST_ec_recovery_multiple_errors: setup_osds 9 2020-09-08T16:58:12.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=9 2020-09-08T16:58:12.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T16:58:12.243 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 9 - 1 2020-09-08T16:58:12.243 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 8 2020-09-08T16:58:12.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:58:12.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T16:58:12.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:58:12.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:58:12.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T16:58:12.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:58:12.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T16:58:12.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:58:12.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:58:12.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:58:12.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:58:12.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T16:58:12.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T16:58:12.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:58:12.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:58:12.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:58:12.246 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:58:12.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:12.247 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:12.247 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:12.247 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:12.247 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:12.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:12.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:12.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:58:12.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:58:12.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:58:12.249 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:58:12.249 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:58:12.249 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:58:12.249 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:58:12.249 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:12.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:58:12.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:58:12.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T16:58:12.250 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:58:12.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6fb1aa43-77a8-4b55-992a-9c08d8e41bdf 2020-09-08T16:58:12.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 6fb1aa43-77a8-4b55-992a-9c08d8e41bdf' 2020-09-08T16:58:12.251 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:58:12.252 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 6fb1aa43-77a8-4b55-992a-9c08d8e41bdf 2020-09-08T16:58:12.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAkuFdfz5abDxAALzeJaerzkXtBqgAAxw+PMw== 2020-09-08T16:58:12.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAkuFdfz5abDxAALzeJaerzkXtBqgAAxw+PMw=="}' 2020-09-08T16:58:12.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6fb1aa43-77a8-4b55-992a-9c08d8e41bdf -i td/test-erasure-eio/0/new.json 2020-09-08T16:58:12.627 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:58:12.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T16:58:12.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAkuFdfz5abDxAALzeJaerzkXtBqgAAxw+PMw== --osd-uuid 6fb1aa43-77a8-4b55-992a-9c08d8e41bdf 2020-09-08T16:58:12.659 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:12.665+0000 7f440800fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:12.661 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:12.667+0000 7f440800fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:12.662 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:12.668+0000 7f440800fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:12.663 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:12.669+0000 7f440800fec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:58:12.663 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:12.669+0000 7f440800fec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:58:12.664 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:12.669+0000 7f440800fec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T16:58:12.730 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:12.736+0000 7f440800fec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T16:58:17.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T16:58:17.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:58:17.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T16:58:17.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:58:17.384 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T16:58:17.874 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T16:58:17.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T16:58:17.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:17.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:58:17.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:58:17.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:58:17.893 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:17.898+0000 7f34011f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:17.894 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:17.900+0000 7f34011f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:17.896 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:17.901+0000 7f34011f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:18.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T16:58:18.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:58:18.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T16:58:18.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:58:18.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:58:18.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:18.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:58:18.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:18.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:58:18.320 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:58:18.416 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:18.422+0000 7f34011f1ec0 -1 Falling back to public interface 2020-09-08T16:58:18.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:58:19.256 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:19.262+0000 7f34011f1ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T16:58:19.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:58:19.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:19.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:58:19.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:19.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T16:58:19.704 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:58:20.091 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/90538,v1:127.0.0.1:6803/90538] [v2:127.0.0.1:6804/90538,v1:127.0.0.1:6805/90538] exists,up 6fb1aa43-77a8-4b55-992a-9c08d8e41bdf 2020-09-08T16:58:20.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:58:20.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:58:20.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:58:20.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:58:20.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T16:58:20.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:58:20.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:58:20.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T16:58:20.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:58:20.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T16:58:20.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:58:20.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:58:20.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:58:20.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:58:20.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T16:58:20.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T16:58:20.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:58:20.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:58:20.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:58:20.096 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:58:20.096 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:20.096 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:20.096 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:20.096 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:20.096 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:20.097 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:20.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:20.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:58:20.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:58:20.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:58:20.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:58:20.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:58:20.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:58:20.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:58:20.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:20.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:58:20.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:58:20.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T16:58:20.099 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:58:20.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=64c52dad-7afc-46fe-9382-1c33192ed364 2020-09-08T16:58:20.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 64c52dad-7afc-46fe-9382-1c33192ed364' 2020-09-08T16:58:20.100 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:58:20.101 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 64c52dad-7afc-46fe-9382-1c33192ed364 2020-09-08T16:58:20.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAsuFdfQD+OBhAACiA+TjWIpgizAL5swqQCaQ== 2020-09-08T16:58:20.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAsuFdfQD+OBhAACiA+TjWIpgizAL5swqQCaQ=="}' 2020-09-08T16:58:20.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 64c52dad-7afc-46fe-9382-1c33192ed364 -i td/test-erasure-eio/1/new.json 2020-09-08T16:58:20.925 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:58:20.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T16:58:20.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAsuFdfQD+OBhAACiA+TjWIpgizAL5swqQCaQ== --osd-uuid 64c52dad-7afc-46fe-9382-1c33192ed364 2020-09-08T16:58:20.956 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:20.961+0000 7fa02fd97ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:20.957 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:20.963+0000 7fa02fd97ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:20.959 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:20.965+0000 7fa02fd97ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:20.960 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:20.965+0000 7fa02fd97ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:58:20.960 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:20.965+0000 7fa02fd97ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:58:20.960 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:20.965+0000 7fa02fd97ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T16:58:21.067 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:21.072+0000 7fa02fd97ec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T16:58:25.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T16:58:25.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:58:25.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T16:58:25.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:58:25.443 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T16:58:25.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T16:58:25.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:25.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:58:25.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:58:25.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:58:25.938 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T16:58:25.955 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:25.960+0000 7f062ad2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:25.957 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:25.962+0000 7f062ad2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:25.958 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:25.963+0000 7f062ad2fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:26.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T16:58:26.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:58:26.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T16:58:26.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:58:26.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:58:26.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:26.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:58:26.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:26.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:58:26.346 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:58:26.478 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:26.482+0000 7f062ad2fec0 -1 Falling back to public interface 2020-09-08T16:58:26.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:58:27.283 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:27.288+0000 7f062ad2fec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T16:58:27.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:58:27.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:27.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:58:27.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T16:58:27.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:27.713 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:58:28.203 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/91532,v1:127.0.0.1:6811/91532] [v2:127.0.0.1:6812/91532,v1:127.0.0.1:6813/91532] exists,up 64c52dad-7afc-46fe-9382-1c33192ed364 2020-09-08T16:58:28.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:58:28.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:58:28.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:58:28.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:58:28.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T16:58:28.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:58:28.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:58:28.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T16:58:28.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:58:28.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T16:58:28.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:58:28.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:58:28.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:58:28.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:58:28.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T16:58:28.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T16:58:28.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:58:28.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:58:28.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:58:28.208 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:58:28.208 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:28.208 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:28.208 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:28.209 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:28.209 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:28.209 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:28.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:28.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:58:28.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:58:28.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:58:28.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:58:28.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:58:28.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:58:28.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:58:28.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:28.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:58:28.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:58:28.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T16:58:28.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:58:28.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ae8d3ce4-0cad-46ee-885f-4f135b45719b 2020-09-08T16:58:28.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 ae8d3ce4-0cad-46ee-885f-4f135b45719b' 2020-09-08T16:58:28.213 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:58:28.213 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 ae8d3ce4-0cad-46ee-885f-4f135b45719b 2020-09-08T16:58:28.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA0uFdf9y40DRAAv8yuSHHIHHv1ECT61G9dGw== 2020-09-08T16:58:28.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA0uFdf9y40DRAAv8yuSHHIHHv1ECT61G9dGw=="}' 2020-09-08T16:58:28.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ae8d3ce4-0cad-46ee-885f-4f135b45719b -i td/test-erasure-eio/2/new.json 2020-09-08T16:58:28.953 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:58:28.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T16:58:28.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA0uFdf9y40DRAAv8yuSHHIHHv1ECT61G9dGw== --osd-uuid ae8d3ce4-0cad-46ee-885f-4f135b45719b 2020-09-08T16:58:28.986 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:28.990+0000 7f595cb73ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:28.987 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:28.992+0000 7f595cb73ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:28.988 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:28.993+0000 7f595cb73ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:28.989 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:28.994+0000 7f595cb73ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:58:28.989 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:28.994+0000 7f595cb73ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:58:28.990 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:28.994+0000 7f595cb73ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T16:58:29.037 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:29.041+0000 7f595cb73ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T16:58:33.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T16:58:33.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:58:33.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T16:58:33.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:58:33.879 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T16:58:34.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T16:58:34.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:58:34.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:58:34.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:58:34.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:34.366 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T16:58:34.383 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:34.388+0000 7f0b4a94dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:34.385 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:34.390+0000 7f0b4a94dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:34.387 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:34.391+0000 7f0b4a94dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:34.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T16:58:34.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:58:34.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T16:58:34.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:58:34.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:58:34.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:34.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:58:34.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:34.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:58:34.782 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:58:34.906 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:34.911+0000 7f0b4a94dec0 -1 Falling back to public interface 2020-09-08T16:58:35.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:58:35.753 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:35.758+0000 7f0b4a94dec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T16:58:36.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:58:36.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:36.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:58:36.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:36.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T16:58:36.169 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:58:36.647 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/92526,v1:127.0.0.1:6819/92526] [v2:127.0.0.1:6820/92526,v1:127.0.0.1:6821/92526] exists,up ae8d3ce4-0cad-46ee-885f-4f135b45719b 2020-09-08T16:58:36.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:58:36.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:58:36.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:58:36.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:58:36.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T16:58:36.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:58:36.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:58:36.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T16:58:36.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:58:36.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T16:58:36.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:58:36.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:58:36.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:58:36.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:58:36.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T16:58:36.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T16:58:36.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:58:36.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:58:36.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:58:36.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:58:36.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:36.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:36.652 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:36.653 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:36.653 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:36.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:36.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:36.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:58:36.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:58:36.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:58:36.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:58:36.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:58:36.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:58:36.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:58:36.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:36.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:58:36.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:58:36.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T16:58:36.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:58:36.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d65bd25f-8b77-43d9-92b4-cea293bdb42e 2020-09-08T16:58:36.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 d65bd25f-8b77-43d9-92b4-cea293bdb42e' 2020-09-08T16:58:36.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:58:36.657 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 d65bd25f-8b77-43d9-92b4-cea293bdb42e 2020-09-08T16:58:36.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA8uFdf8tm0JxAAaElSEz50UBFnuK392vxsjw== 2020-09-08T16:58:36.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA8uFdf8tm0JxAAaElSEz50UBFnuK392vxsjw=="}' 2020-09-08T16:58:36.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d65bd25f-8b77-43d9-92b4-cea293bdb42e -i td/test-erasure-eio/3/new.json 2020-09-08T16:58:37.597 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T16:58:37.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T16:58:37.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8uFdf8tm0JxAAaElSEz50UBFnuK392vxsjw== --osd-uuid d65bd25f-8b77-43d9-92b4-cea293bdb42e 2020-09-08T16:58:37.649 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:37.633+0000 7fac75f86ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:37.652 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:37.656+0000 7fac75f86ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:37.652 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:37.657+0000 7fac75f86ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:37.653 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:37.657+0000 7fac75f86ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:58:37.653 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:37.657+0000 7fac75f86ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:58:37.654 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:37.657+0000 7fac75f86ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T16:58:37.808 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:37.813+0000 7fac75f86ec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T16:58:42.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T16:58:42.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:58:42.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T16:58:42.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:58:42.659 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T16:58:43.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T16:58:43.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:43.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:58:43.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:58:43.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:58:43.164 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T16:58:43.181 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:43.185+0000 7ffb6aa73ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:43.182 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:43.187+0000 7ffb6aa73ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:43.183 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:43.188+0000 7ffb6aa73ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:43.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T16:58:43.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:58:43.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T16:58:43.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:58:43.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:58:43.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:43.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:58:43.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:43.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:58:43.562 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:58:43.699 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:43.703+0000 7ffb6aa73ec0 -1 Falling back to public interface 2020-09-08T16:58:43.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:58:44.595 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:44.600+0000 7ffb6aa73ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T16:58:44.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:58:44.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:44.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:58:44.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:44.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:58:44.958 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:58:45.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:58:46.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:58:46.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:46.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:58:46.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:46.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T16:58:46.456 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:58:46.823 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/93528,v1:127.0.0.1:6827/93528] [v2:127.0.0.1:6828/93528,v1:127.0.0.1:6829/93528] exists,up d65bd25f-8b77-43d9-92b4-cea293bdb42e 2020-09-08T16:58:46.824 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 813bf1e0-c8e3-404a-b5d2-4cc6fa0a48a7 2020-09-08T16:58:46.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:58:46.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:58:46.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:58:46.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:58:46.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 4 2020-09-08T16:58:46.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:58:46.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:58:46.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T16:58:46.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:58:46.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/4 2020-09-08T16:58:46.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:58:46.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:58:46.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:58:46.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:58:46.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2020-09-08T16:58:46.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2020-09-08T16:58:46.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:58:46.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:58:46.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:58:46.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:58:46.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:46.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:46.828 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:46.829 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:46.829 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:46.829 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:46.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:46.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:58:46.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:58:46.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:58:46.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:58:46.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:58:46.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:58:46.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:58:46.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:46.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:58:46.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:58:46.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/4 2020-09-08T16:58:46.831 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:58:46.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=813bf1e0-c8e3-404a-b5d2-4cc6fa0a48a7 2020-09-08T16:58:46.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 813bf1e0-c8e3-404a-b5d2-4cc6fa0a48a7' 2020-09-08T16:58:46.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:58:46.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBGuFdfrJcWMhAAFVgPii6zVcP0Sj0UuI2LOg== 2020-09-08T16:58:46.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBGuFdfrJcWMhAAFVgPii6zVcP0Sj0UuI2LOg=="}' 2020-09-08T16:58:46.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 813bf1e0-c8e3-404a-b5d2-4cc6fa0a48a7 -i td/test-erasure-eio/4/new.json 2020-09-08T16:58:47.436 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T16:58:47.446 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/4/new.json 2020-09-08T16:58:47.446 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBGuFdfrJcWMhAAFVgPii6zVcP0Sj0UuI2LOg== --osd-uuid 813bf1e0-c8e3-404a-b5d2-4cc6fa0a48a7 2020-09-08T16:58:47.467 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:47.472+0000 7fc1c283dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:47.469 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:47.473+0000 7fc1c283dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:47.470 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:47.474+0000 7fc1c283dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:47.470 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:47.475+0000 7fc1c283dec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:58:47.471 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:47.475+0000 7fc1c283dec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:58:47.471 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:47.475+0000 7fc1c283dec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T16:58:47.554 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:47.559+0000 7fc1c283dec0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2020-09-08T16:58:52.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2020-09-08T16:58:52.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:58:52.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T16:58:52.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:58:52.396 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T16:58:52.891 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T16:58:52.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T16:58:52.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:52.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:58:52.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:58:52.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:58:52.910 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:52.915+0000 7f31700ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:52.911 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:52.917+0000 7f31700ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:52.913 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:52.918+0000 7f31700ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:53.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T16:58:53.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:58:53.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T16:58:53.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:58:53.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:58:53.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:53.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:58:53.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:53.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:58:53.277 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:58:53.430 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:53.435+0000 7f31700ffec0 -1 Falling back to public interface 2020-09-08T16:58:53.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:58:54.298 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:54.303+0000 7f31700ffec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T16:58:54.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:58:54.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:58:54.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:58:54.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:58:54.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T16:58:54.639 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:58:55.017 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/94548,v1:127.0.0.1:6835/94548] [v2:127.0.0.1:6836/94548,v1:127.0.0.1:6837/94548] exists,up 813bf1e0-c8e3-404a-b5d2-4cc6fa0a48a7 2020-09-08T16:58:55.017 INFO:tasks.workunit.client.0.smithi120.stdout:add osd5 072bc03d-42e0-46d5-a569-2db68a991c41 2020-09-08T16:58:55.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:58:55.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:58:55.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:58:55.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:58:55.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 5 2020-09-08T16:58:55.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:58:55.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:58:55.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-09-08T16:58:55.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:58:55.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/5 2020-09-08T16:58:55.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:58:55.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:58:55.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:58:55.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:58:55.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2020-09-08T16:58:55.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2020-09-08T16:58:55.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:58:55.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:58:55.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:58:55.021 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:58:55.022 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:58:55.022 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:58:55.022 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:58:55.022 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:58:55.022 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:58:55.023 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:55.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:58:55.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:58:55.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:58:55.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:58:55.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:58:55.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:58:55.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:58:55.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:58:55.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:58:55.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:58:55.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:58:55.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/5 2020-09-08T16:58:55.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:58:55.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=072bc03d-42e0-46d5-a569-2db68a991c41 2020-09-08T16:58:55.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 072bc03d-42e0-46d5-a569-2db68a991c41' 2020-09-08T16:58:55.026 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:58:55.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBPuFdfPIERAhAArmhwTuKUnnPpeUUfc2duwg== 2020-09-08T16:58:55.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBPuFdfPIERAhAArmhwTuKUnnPpeUUfc2duwg=="}' 2020-09-08T16:58:55.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 072bc03d-42e0-46d5-a569-2db68a991c41 -i td/test-erasure-eio/5/new.json 2020-09-08T16:58:55.999 INFO:tasks.workunit.client.0.smithi120.stdout:5 2020-09-08T16:58:56.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/5/new.json 2020-09-08T16:58:56.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBPuFdfPIERAhAArmhwTuKUnnPpeUUfc2duwg== --osd-uuid 072bc03d-42e0-46d5-a569-2db68a991c41 2020-09-08T16:58:56.030 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:56.036+0000 7fdac1fbeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:56.057 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:56.062+0000 7fdac1fbeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:56.058 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:56.063+0000 7fdac1fbeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:58:56.058 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:56.064+0000 7fdac1fbeec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T16:58:56.059 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:56.064+0000 7fdac1fbeec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T16:58:56.059 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:56.064+0000 7fdac1fbeec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T16:58:56.191 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:58:56.196+0000 7fdac1fbeec0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2020-09-08T16:59:01.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2020-09-08T16:59:01.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:59:01.251 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd5 key to auth repository 2020-09-08T16:59:01.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-09-08T16:59:01.252 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:59:01.749 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.5 2020-09-08T16:59:01.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-09-08T16:59:01.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:59:01.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:59:01.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:59:01.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:59:01.768 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:01.773+0000 7fecfae7cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:01.770 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:01.775+0000 7fecfae7cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:01.771 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:01.777+0000 7fecfae7cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:02.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-09-08T16:59:02.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:59:02.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-09-08T16:59:02.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:59:02.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:59:02.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:02.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:59:02.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:02.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:59:02.114 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:59:02.287 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:02.292+0000 7fecfae7cec0 -1 Falling back to public interface 2020-09-08T16:59:02.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:03.167 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:03.172+0000 7fecfae7cec0 -1 osd.5 0 log_to_monitors {default=true} 2020-09-08T16:59:03.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:03.491 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:03.491 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:59:03.491 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:03.491 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:59:03.492 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:59:03.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:04.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:04.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:04.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:59:04.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:04.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T16:59:04.871 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:59:05.271 INFO:tasks.workunit.client.0.smithi120.stdout:osd.5 up in weight 1 up_from 40 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/95542,v1:127.0.0.1:6843/95542] [v2:127.0.0.1:6844/95542,v1:127.0.0.1:6845/95542] exists,up 072bc03d-42e0-46d5-a569-2db68a991c41 2020-09-08T16:59:05.271 INFO:tasks.workunit.client.0.smithi120.stdout:add osd6 9de0504b-1626-484a-b664-547a8ac17157 2020-09-08T16:59:05.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:59:05.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:59:05.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:59:05.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:59:05.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 6 2020-09-08T16:59:05.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:59:05.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:59:05.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2020-09-08T16:59:05.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:59:05.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/6 2020-09-08T16:59:05.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:59:05.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:59:05.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:59:05.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:59:05.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2020-09-08T16:59:05.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2020-09-08T16:59:05.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:59:05.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:59:05.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:59:05.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:59:05.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:59:05.276 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:59:05.276 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:59:05.276 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:05.276 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:05.277 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:59:05.277 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:59:05.277 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:59:05.277 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:59:05.277 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:59:05.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:59:05.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:59:05.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:59:05.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:59:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:59:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:59:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:59:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/6 2020-09-08T16:59:05.279 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:59:05.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9de0504b-1626-484a-b664-547a8ac17157 2020-09-08T16:59:05.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 9de0504b-1626-484a-b664-547a8ac17157' 2020-09-08T16:59:05.280 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:59:05.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBZuFdfiuEwERAAM89nlG9a24BoSfhu52SHNA== 2020-09-08T16:59:05.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBZuFdfiuEwERAAM89nlG9a24BoSfhu52SHNA=="}' 2020-09-08T16:59:05.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9de0504b-1626-484a-b664-547a8ac17157 -i td/test-erasure-eio/6/new.json 2020-09-08T16:59:05.899 INFO:tasks.workunit.client.0.smithi120.stdout:6 2020-09-08T16:59:05.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/6/new.json 2020-09-08T16:59:05.910 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZuFdfiuEwERAAM89nlG9a24BoSfhu52SHNA== --osd-uuid 9de0504b-1626-484a-b664-547a8ac17157 2020-09-08T16:59:05.930 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:05.936+0000 7f75d46cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:05.932 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:05.938+0000 7f75d46cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:05.934 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:05.939+0000 7f75d46cdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:05.934 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:05.940+0000 7f75d46cdec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T16:59:05.935 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:05.940+0000 7f75d46cdec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T16:59:05.935 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:05.940+0000 7f75d46cdec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T16:59:06.060 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:06.066+0000 7f75d46cdec0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2020-09-08T16:59:10.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2020-09-08T16:59:10.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:59:10.781 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd6 key to auth repository 2020-09-08T16:59:10.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2020-09-08T16:59:10.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:59:11.285 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.6 2020-09-08T16:59:11.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2020-09-08T16:59:11.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:59:11.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:59:11.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:59:11.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:59:11.304 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:11.308+0000 7f5852d52ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:11.305 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:11.310+0000 7f5852d52ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:11.307 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:11.312+0000 7f5852d52ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:11.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2020-09-08T16:59:11.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:59:11.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-09-08T16:59:11.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:59:11.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:59:11.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:11.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:59:11.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:59:11.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:11.658 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:59:11.823 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:11.828+0000 7f5852d52ec0 -1 Falling back to public interface 2020-09-08T16:59:12.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:12.710 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:12.714+0000 7f5852d52ec0 -1 osd.6 0 log_to_monitors {default=true} 2020-09-08T16:59:13.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:13.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:13.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:59:13.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:13.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:59:13.039 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:59:13.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:14.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:14.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:14.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:59:14.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:14.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T16:59:14.461 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:59:14.842 INFO:tasks.workunit.client.0.smithi120.stdout:osd.6 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/96562,v1:127.0.0.1:6851/96562] [v2:127.0.0.1:6852/96562,v1:127.0.0.1:6853/96562] exists,up 9de0504b-1626-484a-b664-547a8ac17157 2020-09-08T16:59:14.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:59:14.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:59:14.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:59:14.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:59:14.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 7 2020-09-08T16:59:14.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:59:14.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:59:14.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=7 2020-09-08T16:59:14.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:59:14.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/7 2020-09-08T16:59:14.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:59:14.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:59:14.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:59:14.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:59:14.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2020-09-08T16:59:14.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2020-09-08T16:59:14.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:59:14.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:59:14.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:59:14.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:59:14.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:59:14.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:59:14.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:59:14.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:14.847 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:14.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:59:14.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:59:14.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:59:14.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:59:14.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:59:14.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:59:14.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:59:14.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:59:14.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:59:14.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:59:14.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:59:14.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:59:14.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/7 2020-09-08T16:59:14.850 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:59:14.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=273a2e7d-1248-418d-8e73-fbba4a2c1ba0 2020-09-08T16:59:14.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd7 273a2e7d-1248-418d-8e73-fbba4a2c1ba0' 2020-09-08T16:59:14.851 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:59:14.851 INFO:tasks.workunit.client.0.smithi120.stdout:add osd7 273a2e7d-1248-418d-8e73-fbba4a2c1ba0 2020-09-08T16:59:14.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBiuFdfQzhKMxAASzcdYZLznF0vFNCmnnzm/w== 2020-09-08T16:59:14.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBiuFdfQzhKMxAASzcdYZLznF0vFNCmnnzm/w=="}' 2020-09-08T16:59:14.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 273a2e7d-1248-418d-8e73-fbba4a2c1ba0 -i td/test-erasure-eio/7/new.json 2020-09-08T16:59:15.621 INFO:tasks.workunit.client.0.smithi120.stdout:7 2020-09-08T16:59:15.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/7/new.json 2020-09-08T16:59:15.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 7 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBiuFdfQzhKMxAASzcdYZLznF0vFNCmnnzm/w== --osd-uuid 273a2e7d-1248-418d-8e73-fbba4a2c1ba0 2020-09-08T16:59:15.652 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:15.657+0000 7f6a6988dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:15.654 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:15.658+0000 7f6a6988dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:15.654 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:15.659+0000 7f6a6988dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:15.655 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:15.660+0000 7f6a6988dec0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2020-09-08T16:59:15.655 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:15.660+0000 7f6a6988dec0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2020-09-08T16:59:15.656 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:15.660+0000 7f6a6988dec0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2020-09-08T16:59:15.709 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:15.714+0000 7f6a6988dec0 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2020-09-08T16:59:21.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2020-09-08T16:59:21.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:59:21.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd7 key to auth repository 2020-09-08T16:59:21.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:59:21.012 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd7 key to auth repository 2020-09-08T16:59:21.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.7 2020-09-08T16:59:21.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 7 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:59:21.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:59:21.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:59:21.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:59:21.512 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.7 2020-09-08T16:59:21.529 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:21.534+0000 7f17bf678ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:21.531 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:21.535+0000 7f17bf678ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:21.532 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:21.537+0000 7f17bf678ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:21.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 7 2020-09-08T16:59:21.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:59:21.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=7 2020-09-08T16:59:21.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:59:21.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:59:21.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:21.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:59:21.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:21.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2020-09-08T16:59:21.879 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:59:22.048 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:22.053+0000 7f17bf678ec0 -1 Falling back to public interface 2020-09-08T16:59:22.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:22.984 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:22.989+0000 7f17bf678ec0 -1 osd.7 0 log_to_monitors {default=true} 2020-09-08T16:59:23.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:23.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:23.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:59:23.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:23.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2020-09-08T16:59:23.246 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:59:23.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:24.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:24.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:24.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:59:24.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:24.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2020-09-08T16:59:24.689 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:59:25.146 INFO:tasks.workunit.client.0.smithi120.stdout:osd.7 up in weight 1 up_from 53 up_thru 53 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/97582,v1:127.0.0.1:6859/97582] [v2:127.0.0.1:6860/97582,v1:127.0.0.1:6861/97582] exists,up 273a2e7d-1248-418d-8e73-fbba4a2c1ba0 2020-09-08T16:59:25.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:59:25.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:59:25.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:59:25.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T16:59:25.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 8 2020-09-08T16:59:25.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T16:59:25.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T16:59:25.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=8 2020-09-08T16:59:25.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T16:59:25.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/8 2020-09-08T16:59:25.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T16:59:25.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T16:59:25.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T16:59:25.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T16:59:25.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2020-09-08T16:59:25.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2020-09-08T16:59:25.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T16:59:25.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T16:59:25.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T16:59:25.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T16:59:25.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:59:25.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:59:25.152 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:59:25.152 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:25.152 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:25.152 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:59:25.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:59:25.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T16:59:25.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T16:59:25.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T16:59:25.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T16:59:25.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T16:59:25.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T16:59:25.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T16:59:25.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:59:25.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T16:59:25.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T16:59:25.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/8 2020-09-08T16:59:25.155 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T16:59:25.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dc90a3a7-f00a-41e1-b442-27fcc8301fc9 2020-09-08T16:59:25.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd8 dc90a3a7-f00a-41e1-b442-27fcc8301fc9' 2020-09-08T16:59:25.157 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T16:59:25.157 INFO:tasks.workunit.client.0.smithi120.stdout:add osd8 dc90a3a7-f00a-41e1-b442-27fcc8301fc9 2020-09-08T16:59:25.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBtuFdfYQngCRAAGUFQ9lf6wHXUF30qejAv/Q== 2020-09-08T16:59:25.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBtuFdfYQngCRAAGUFQ9lf6wHXUF30qejAv/Q=="}' 2020-09-08T16:59:25.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dc90a3a7-f00a-41e1-b442-27fcc8301fc9 -i td/test-erasure-eio/8/new.json 2020-09-08T16:59:25.898 INFO:tasks.workunit.client.0.smithi120.stdout:8 2020-09-08T16:59:25.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/8/new.json 2020-09-08T16:59:25.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 8 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBtuFdfYQngCRAAGUFQ9lf6wHXUF30qejAv/Q== --osd-uuid dc90a3a7-f00a-41e1-b442-27fcc8301fc9 2020-09-08T16:59:25.930 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:25.935+0000 7f5cd8f6aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:25.932 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:25.936+0000 7f5cd8f6aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:25.932 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:25.937+0000 7f5cd8f6aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:25.933 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:25.937+0000 7f5cd8f6aec0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2020-09-08T16:59:25.933 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:25.937+0000 7f5cd8f6aec0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2020-09-08T16:59:25.933 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:25.937+0000 7f5cd8f6aec0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2020-09-08T16:59:26.143 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:26.148+0000 7f5cd8f6aec0 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2020-09-08T16:59:31.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2020-09-08T16:59:31.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T16:59:31.317 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd8 key to auth repository 2020-09-08T16:59:31.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd8 key to auth repository 2020-09-08T16:59:31.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T16:59:31.815 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.8 2020-09-08T16:59:31.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.8 2020-09-08T16:59:31.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 8 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T16:59:31.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T16:59:31.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T16:59:31.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T16:59:31.833 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:31.838+0000 7fa2b3b95ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:31.835 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:31.840+0000 7fa2b3b95ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:31.836 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:31.841+0000 7fa2b3b95ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T16:59:32.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 8 2020-09-08T16:59:32.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T16:59:32.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=8 2020-09-08T16:59:32.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T16:59:32.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T16:59:32.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:32.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T16:59:32.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:32.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2020-09-08T16:59:32.273 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T16:59:32.356 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:32.360+0000 7fa2b3b95ec0 -1 Falling back to public interface 2020-09-08T16:59:32.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:33.276 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T16:59:33.281+0000 7fa2b3b95ec0 -1 osd.8 0 log_to_monitors {default=true} 2020-09-08T16:59:33.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:33.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:33.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T16:59:33.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:33.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2020-09-08T16:59:33.557 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T16:59:34.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T16:59:35.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T16:59:35.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T16:59:35.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T16:59:35.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T16:59:35.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2020-09-08T16:59:35.015 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T16:59:35.523 INFO:tasks.workunit.client.0.smithi120.stdout:osd.8 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/98602,v1:127.0.0.1:6867/98602] [v2:127.0.0.1:6868/98602,v1:127.0.0.1:6869/98602] exists,up dc90a3a7-f00a-41e1-b442-27fcc8301fc9 2020-09-08T16:59:35.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T16:59:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T16:59:35.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T16:59:35.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T16:59:35.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:59:35.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:59:35.525 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:35.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T16:59:35.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T16:59:35.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T16:59:35.614 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T16:59:35.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T16:59:35.625 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:58:18.430+0000 7f34011f1ec0 0 load: jerasure load: lrc load: isa 2020-09-08T16:59:35.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:422: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2020-09-08T16:59:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2020-09-08T16:59:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T16:59:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T16:59:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=4 2020-09-08T16:59:35.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T16:59:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=4 2020-09-08T16:59:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T16:59:35.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2020-09-08T16:59:36.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T16:59:36.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T16:59:37.707 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T16:59:37.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T16:59:38.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T16:59:38.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:59:38.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:59:38.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:59:38.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:59:38.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:59:38.719 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:59:38.719 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:59:38.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:59:38.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:59:38.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:59:38.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:59:38.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:59:38.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:59:38.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:59:38.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:59:39.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:59:39.201 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:59:39.201 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:59:39.202 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:59:39.202 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:59:39.202 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:59:39.202 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:59:39.202 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:59:39.203 INFO:tasks.workunit.client.0.smithi120.stderr:8' 2020-09-08T16:59:39.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:59:39.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:39.203 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:59:39.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803795 2020-09-08T16:59:39.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803795 2020-09-08T16:59:39.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795' 2020-09-08T16:59:39.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:39.323 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:59:39.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574865 2020-09-08T16:59:39.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574865 2020-09-08T16:59:39.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865' 2020-09-08T16:59:39.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:39.463 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:59:39.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378639 2020-09-08T16:59:39.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378639 2020-09-08T16:59:39.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865 2-81604378639' 2020-09-08T16:59:39.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:39.584 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:59:39.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149710 2020-09-08T16:59:39.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149710 2020-09-08T16:59:39.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865 2-81604378639 3-111669149710' 2020-09-08T16:59:39.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:39.704 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:59:39.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920779 2020-09-08T16:59:39.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920779 2020-09-08T16:59:39.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865 2-81604378639 3-111669149710 4-141733920779' 2020-09-08T16:59:39.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:39.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:59:40.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691850 2020-09-08T16:59:40.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691850 2020-09-08T16:59:40.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865 2-81604378639 3-111669149710 4-141733920779 5-171798691850' 2020-09-08T16:59:40.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:40.078 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:59:40.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495625 2020-09-08T16:59:40.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495625 2020-09-08T16:59:40.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865 2-81604378639 3-111669149710 4-141733920779 5-171798691850 6-197568495625' 2020-09-08T16:59:40.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:40.199 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:59:40.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=227633266694 2020-09-08T16:59:40.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 227633266694 2020-09-08T16:59:40.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865 2-81604378639 3-111669149710 4-141733920779 5-171798691850 6-197568495625 7-227633266694' 2020-09-08T16:59:40.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:40.321 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:59:40.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=253403070468 2020-09-08T16:59:40.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 253403070468 2020-09-08T16:59:40.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574865 2-81604378639 3-111669149710 4-141733920779 5-171798691850 6-197568495625 7-227633266694 8-253403070468' 2020-09-08T16:59:40.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:40.443 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803795 2020-09-08T16:59:40.443 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:40.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:59:40.444 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803795 2020-09-08T16:59:40.444 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:40.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803795 2020-09-08T16:59:40.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803795' 2020-09-08T16:59:40.445 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:59:40.446 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803795 2020-09-08T16:59:40.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803795 -lt 25769803795 2020-09-08T16:59:40.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:40.807 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574865 2020-09-08T16:59:40.807 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:40.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:59:40.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574865 2020-09-08T16:59:40.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:40.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574865 2020-09-08T16:59:40.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574865' 2020-09-08T16:59:40.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:59:40.810 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574865 2020-09-08T16:59:41.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574865 -lt 55834574865 2020-09-08T16:59:41.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:41.235 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378639 2020-09-08T16:59:41.235 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:41.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:59:41.236 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378639 2020-09-08T16:59:41.236 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:41.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378639 2020-09-08T16:59:41.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2020-09-08T16:59:41.237 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:59:41.237 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378639 2020-09-08T16:59:41.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378639 -lt 81604378639 2020-09-08T16:59:41.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:41.605 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149710 2020-09-08T16:59:41.605 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:41.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:59:41.606 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149710 2020-09-08T16:59:41.606 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:41.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149710 2020-09-08T16:59:41.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149710' 2020-09-08T16:59:41.607 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:59:41.607 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149710 2020-09-08T16:59:41.963 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149710 -lt 111669149710 2020-09-08T16:59:41.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:41.964 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920779 2020-09-08T16:59:41.965 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:41.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:59:41.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920779 2020-09-08T16:59:41.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:41.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920779 2020-09-08T16:59:41.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920779' 2020-09-08T16:59:41.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:59:41.967 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920779 2020-09-08T16:59:42.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920779 -lt 141733920779 2020-09-08T16:59:42.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:42.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-171798691850 2020-09-08T16:59:42.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:42.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:59:42.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-171798691850 2020-09-08T16:59:42.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:42.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691850 2020-09-08T16:59:42.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 171798691850' 2020-09-08T16:59:42.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:59:42.335 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 171798691850 2020-09-08T16:59:42.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691850 -lt 171798691850 2020-09-08T16:59:42.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:42.720 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-197568495625 2020-09-08T16:59:42.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:42.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:59:42.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-197568495625 2020-09-08T16:59:42.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:42.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495625 2020-09-08T16:59:42.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 197568495625' 2020-09-08T16:59:42.723 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:59:42.723 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 197568495625 2020-09-08T16:59:43.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495625 -lt 197568495625 2020-09-08T16:59:43.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:43.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-227633266694 2020-09-08T16:59:43.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:43.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2020-09-08T16:59:43.087 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-227633266694 2020-09-08T16:59:43.087 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:43.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=227633266694 2020-09-08T16:59:43.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 227633266694' 2020-09-08T16:59:43.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2020-09-08T16:59:43.089 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.7 seq 227633266694 2020-09-08T16:59:43.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 227633266694 -lt 227633266694 2020-09-08T16:59:43.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:43.455 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-253403070468 2020-09-08T16:59:43.455 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:43.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:59:43.456 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-253403070468 2020-09-08T16:59:43.456 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:43.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=253403070468 2020-09-08T16:59:43.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 253403070468' 2020-09-08T16:59:43.457 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:59:43.458 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.8 seq 253403070468 2020-09-08T16:59:43.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 253403070468 -lt 253403070468 2020-09-08T16:59:43.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:59:43.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:59:43.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:59:44.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T16:59:44.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:59:44.343 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:59:44.343 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:59:44.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:59:44.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:59:44.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:59:44.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:59:44.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T16:59:44.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:59:44.702 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:59:44.702 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:59:45.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T16:59:45.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:59:45.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:59:45.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:425: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2020-09-08T16:59:45.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T16:59:45.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T16:59:45.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=myobject 2020-09-08T16:59:45.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:59:45.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T16:59:45.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:59:45.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T16:59:45.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:59:45.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T16:59:45.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T16:59:45.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T16:59:45.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2020-09-08T16:59:45.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2020-09-08T16:59:45.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T16:59:45.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T16:59:45.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T16:59:45.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T16:59:45.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T16:59:45.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T16:59:45.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=myobject 2020-09-08T16:59:45.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T16:59:45.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T16:59:45.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T16:59:45.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2020-09-08T16:59:45.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T16:59:45.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:59:45.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure myobject 2020-09-08T16:59:45.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:59:45.571 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T16:59:45.571 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T16:59:45.572 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:59:45.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:59:45.936 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:59:45.936 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:59:45.936 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:59:45.936 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:59:45.936 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:59:45.937 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:59:45.937 INFO:tasks.workunit.client.0.smithi120.stderr:7' 2020-09-08T16:59:45.937 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 1 0 6 2 4 7 2020-09-08T16:59:45.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T16:59:45.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2020-09-08T16:59:45.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T16:59:45.938 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/3/type 2020-09-08T16:59:45.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T16:59:45.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2020-09-08T16:59:45.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T16:59:45.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2020-09-08T16:59:45.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T16:59:45.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T16:59:45.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T16:59:45.941 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2020-09-08T16:59:45.941 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T16:59:45.941 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T16:59:45.942 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:45.942 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:45.942 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:45.942 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T16:59:45.942 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2020-09-08T16:59:46.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T16:59:46.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T16:59:46.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T16:59:46.034 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2020-09-08T16:59:46.035 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T16:59:46.035 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T16:59:46.035 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:46.035 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:46.035 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:46.036 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T16:59:46.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T16:59:46.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2020-09-08T16:59:46.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:429: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2020-09-08T16:59:46.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T16:59:46.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T16:59:46.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T16:59:46.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T16:59:46.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T16:59:46.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T16:59:46.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=myobject 2020-09-08T16:59:46.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T16:59:46.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T16:59:46.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T16:59:46.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=3 2020-09-08T16:59:46.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T16:59:46.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:59:46.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure myobject 2020-09-08T16:59:46.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:59:46.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T16:59:46.136 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T16:59:46.136 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:59:46.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:59:46.502 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:59:46.502 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:59:46.502 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:59:46.502 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:59:46.503 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:59:46.503 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:59:46.503 INFO:tasks.workunit.client.0.smithi120.stderr:7' 2020-09-08T16:59:46.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 1 0 6 2 4 7 2020-09-08T16:59:46.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T16:59:46.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=0 2020-09-08T16:59:46.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T16:59:46.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/0/type 2020-09-08T16:59:46.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T16:59:46.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2020-09-08T16:59:46.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T16:59:46.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=0 2020-09-08T16:59:46.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T16:59:46.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T16:59:46.507 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T16:59:46.507 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.0 2020-09-08T16:59:46.507 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:59:46.508 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:59:46.508 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:46.508 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:46.508 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:46.508 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T16:59:46.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2020-09-08T16:59:46.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T16:59:46.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T16:59:46.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T16:59:46.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.0 2020-09-08T16:59:46.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T16:59:46.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T16:59:46.601 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:46.601 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:46.601 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:46.602 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T16:59:46.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T16:59:46.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2020-09-08T16:59:46.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2020-09-08T16:59:46.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T16:59:46.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T16:59:46.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T16:59:46.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T16:59:46.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T16:59:46.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T16:59:46.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=myobject 2020-09-08T16:59:46.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T16:59:46.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T16:59:46.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T16:59:46.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=4 2020-09-08T16:59:46.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T16:59:46.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:59:46.699 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure myobject 2020-09-08T16:59:46.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:59:46.700 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T16:59:46.700 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T16:59:46.700 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:59:47.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:59:47.066 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:59:47.066 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:59:47.066 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:59:47.066 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:59:47.067 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:59:47.067 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:59:47.067 INFO:tasks.workunit.client.0.smithi120.stderr:7' 2020-09-08T16:59:47.067 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 1 0 6 2 4 7 2020-09-08T16:59:47.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T16:59:47.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=6 2020-09-08T16:59:47.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T16:59:47.068 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/6/type 2020-09-08T16:59:47.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T16:59:47.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2020-09-08T16:59:47.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T16:59:47.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=6 2020-09-08T16:59:47.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T16:59:47.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T16:59:47.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T16:59:47.071 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.6 2020-09-08T16:59:47.071 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2020-09-08T16:59:47.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2020-09-08T16:59:47.072 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:47.072 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:47.072 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:47.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.6.asok 2020-09-08T16:59:47.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2020-09-08T16:59:47.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T16:59:47.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T16:59:47.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T16:59:47.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.6 2020-09-08T16:59:47.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2020-09-08T16:59:47.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2020-09-08T16:59:47.165 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:47.165 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:47.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:47.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.6.asok 2020-09-08T16:59:47.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T16:59:47.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2020-09-08T16:59:47.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:432: TEST_ec_recovery_multiple_errors: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T16:59:47.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:432: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2020-09-08T16:59:47.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T16:59:47.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T16:59:47.262 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T16:59:47.262 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T16:59:47.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T16:59:47.631 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:59:47.631 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:59:47.632 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T16:59:47.632 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:59:47.632 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:59:47.632 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:59:47.632 INFO:tasks.workunit.client.0.smithi120.stderr:7' 2020-09-08T16:59:47.633 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 1 0 6 2 4 7 2020-09-08T16:59:47.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:432: TEST_ec_recovery_multiple_errors: local -a initial_osds 2020-09-08T16:59:47.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local last_osd=7 2020-09-08T16:59:47.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:435: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2020-09-08T16:59:47.633 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:59:47.634 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:59:47.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:59:47.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:59:47.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:59:47.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:59:47.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: ceph osd down 7 2020-09-08T16:59:48.197 INFO:tasks.workunit.client.0.smithi120.stderr:osd.7 is already down. 2020-09-08T16:59:48.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd out 7 2020-09-08T16:59:49.093 INFO:tasks.workunit.client.0.smithi120.stderr:osd.7 is already out. 2020-09-08T16:59:49.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:440: TEST_ec_recovery_multiple_errors: wait_for_clean 2020-09-08T16:59:49.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T16:59:49.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T16:59:49.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T16:59:49.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T16:59:49.102 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T16:59:49.103 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T16:59:49.103 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T16:59:49.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T16:59:49.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T16:59:49.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T16:59:49.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T16:59:49.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T16:59:49.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T16:59:49.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T16:59:49.193 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T16:59:49.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T16:59:49.561 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T16:59:49.561 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T16:59:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T16:59:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T16:59:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T16:59:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:6 2020-09-08T16:59:49.562 INFO:tasks.workunit.client.0.smithi120.stderr:7 2020-09-08T16:59:49.563 INFO:tasks.workunit.client.0.smithi120.stderr:8' 2020-09-08T16:59:49.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T16:59:49.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:49.563 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T16:59:49.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803799 2020-09-08T16:59:49.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803799 2020-09-08T16:59:49.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799' 2020-09-08T16:59:49.805 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:49.805 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T16:59:50.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574869 2020-09-08T16:59:50.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574869 2020-09-08T16:59:50.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-55834574869' 2020-09-08T16:59:50.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:50.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T16:59:50.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378643 2020-09-08T16:59:50.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378643 2020-09-08T16:59:50.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-55834574869 2-81604378643' 2020-09-08T16:59:50.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:50.135 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T16:59:50.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149714 2020-09-08T16:59:50.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149714 2020-09-08T16:59:50.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-55834574869 2-81604378643 3-111669149714' 2020-09-08T16:59:50.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:50.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T16:59:50.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920783 2020-09-08T16:59:50.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920783 2020-09-08T16:59:50.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-55834574869 2-81604378643 3-111669149714 4-141733920783' 2020-09-08T16:59:50.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:50.436 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T16:59:50.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691854 2020-09-08T16:59:50.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691854 2020-09-08T16:59:50.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-55834574869 2-81604378643 3-111669149714 4-141733920783 5-171798691854' 2020-09-08T16:59:50.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:50.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T16:59:50.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495629 2020-09-08T16:59:50.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495629 2020-09-08T16:59:50.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-55834574869 2-81604378643 3-111669149714 4-141733920783 5-171798691854 6-197568495629' 2020-09-08T16:59:50.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:50.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2020-09-08T16:59:50.823 INFO:tasks.workunit.client.0.smithi120.stderr:Error ENXIO: problem getting command descriptions from osd.7 2020-09-08T16:59:50.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-09-08T16:59:50.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-09-08T16:59:50.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-09-08T16:59:50.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T16:59:50.825 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2020-09-08T16:59:51.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=253403070472 2020-09-08T16:59:51.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 253403070472 2020-09-08T16:59:51.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-55834574869 2-81604378643 3-111669149714 4-141733920783 5-171798691854 6-197568495629 8-253403070472' 2020-09-08T16:59:51.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:51.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803799 2020-09-08T16:59:51.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:51.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T16:59:51.087 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803799 2020-09-08T16:59:51.087 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:51.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803799 2020-09-08T16:59:51.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803799' 2020-09-08T16:59:51.088 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803799 2020-09-08T16:59:51.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T16:59:51.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803799 -lt 25769803799 2020-09-08T16:59:51.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:51.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574869 2020-09-08T16:59:51.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:51.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T16:59:51.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574869 2020-09-08T16:59:51.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:51.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574869 2020-09-08T16:59:51.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574869' 2020-09-08T16:59:51.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T16:59:51.601 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574869 2020-09-08T16:59:51.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574869 -lt 55834574869 2020-09-08T16:59:51.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:51.965 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378643 2020-09-08T16:59:51.965 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:51.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T16:59:51.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378643 2020-09-08T16:59:51.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:51.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378643 2020-09-08T16:59:51.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378643' 2020-09-08T16:59:51.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T16:59:51.968 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378643 2020-09-08T16:59:52.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378643 -lt 81604378643 2020-09-08T16:59:52.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:52.331 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149714 2020-09-08T16:59:52.331 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:52.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T16:59:52.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149714 2020-09-08T16:59:52.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149714 2020-09-08T16:59:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149714' 2020-09-08T16:59:52.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T16:59:52.333 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149714 2020-09-08T16:59:52.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149715 -lt 111669149714 2020-09-08T16:59:52.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:52.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920783 2020-09-08T16:59:52.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:52.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T16:59:52.703 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920783 2020-09-08T16:59:52.703 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:52.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920783 2020-09-08T16:59:52.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920783' 2020-09-08T16:59:52.704 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T16:59:52.704 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920783 2020-09-08T16:59:53.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920784 -lt 141733920783 2020-09-08T16:59:53.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:53.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-171798691854 2020-09-08T16:59:53.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:53.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T16:59:53.066 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-171798691854 2020-09-08T16:59:53.066 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:53.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691854 2020-09-08T16:59:53.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 171798691854' 2020-09-08T16:59:53.067 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T16:59:53.067 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 171798691854 2020-09-08T16:59:53.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691854 -lt 171798691854 2020-09-08T16:59:53.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:53.450 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-197568495629 2020-09-08T16:59:53.450 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:53.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T16:59:53.451 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-197568495629 2020-09-08T16:59:53.451 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:53.452 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495629 2020-09-08T16:59:53.452 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 197568495629' 2020-09-08T16:59:53.452 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T16:59:53.453 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 197568495629 2020-09-08T16:59:53.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495629 -lt 197568495629 2020-09-08T16:59:53.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T16:59:53.816 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-253403070472 2020-09-08T16:59:53.816 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T16:59:53.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2020-09-08T16:59:53.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-253403070472 2020-09-08T16:59:53.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T16:59:53.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=253403070472 2020-09-08T16:59:53.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 253403070472' 2020-09-08T16:59:53.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2020-09-08T16:59:53.819 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.8 seq 253403070472 2020-09-08T16:59:54.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 253403070472 -lt 253403070472 2020-09-08T16:59:54.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T16:59:54.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:59:54.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:59:54.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T16:59:54.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:59:54.694 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:59:54.694 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:59:54.694 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:59:54.694 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:59:54.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:59:54.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:59:55.079 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:59:55.079 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:59:55.080 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:59:55.080 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:59:55.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:59:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' -1 2020-09-08T16:59:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T16:59:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=4 2020-09-08T16:59:55.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:59:55.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:59:55.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:59:55.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:59:55.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:59:55.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:59:55.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:59:55.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:59:55.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:59:55.661 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:59:56.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T16:59:56.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:59:56.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:59:56.016 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:59:56.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T16:59:56.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T16:59:56.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T16:59:56.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T16:59:56.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T16:59:56.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T16:59:56.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T16:59:56.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T16:59:56.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T16:59:56.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=523 2020-09-08T16:59:57.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 523 '!=' null 2020-09-08T16:59:57.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-09-08T16:59:57.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T16:59:57.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T16:59:57.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T16:59:57.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T16:59:57.101 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T16:59:57.101 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T16:59:57.101 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T16:59:57.102 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T16:59:57.102 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T16:59:57.102 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T16:59:57.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T16:59:57.454 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T16:59:57.455 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T16:59:57.455 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T16:59:57.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T16:59:57.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T16:59:57.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T16:59:57.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:442: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2020-09-08T16:59:57.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T16:59:57.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T16:59:57.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=myobject 2020-09-08T16:59:57.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T16:59:57.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T16:59:57.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2020-09-08T16:59:57.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T16:59:57.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T16:59:57.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:444: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2020-09-08T16:59:57.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T16:59:57.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T16:59:58.460 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T16:59:58.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T16:59:58.962 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T16:59:58.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T16:59:58.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T16:59:58.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:59:58.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T16:59:58.972 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:59:58.972 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:59:58.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:59:58.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:59:58.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:59:59.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:59:59.083 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:59:59.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:59:59.084 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:59:59.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:59:59.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:59:59.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:59:59.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:59:59.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:59:59.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:59:59.087 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:59:59.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:59:59.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:59:59.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T16:59:59.100 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:59:59.100 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:59.100 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:59.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T16:59:59.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:59:59.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:59:59.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T16:59:59.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T16:59:59.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T16:59:59.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T16:59:59.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T16:59:59.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T16:59:59.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T16:59:59.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T16:59:59.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T16:59:59.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T16:59:59.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T16:59:59.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T16:59:59.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T16:59:59.106 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T16:59:59.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T16:59:59.106 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T16:59:59.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T16:59:59.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T16:59:59.107 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T16:59:59.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:59:59.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T16:59:59.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T16:59:59.109 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T16:59:59.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T16:59:59.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T16:59:59.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T16:59:59.111 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T16:59:59.111 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:59.112 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:59.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T16:59:59.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T16:59:59.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T16:59:59.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T16:59:59.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T16:59:59.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:59.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:59.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T16:59:59.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T16:59:59.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T16:59:59.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T16:59:59.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T16:59:59.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T16:59:59.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T16:59:59.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T16:59:59.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T16:59:59.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T16:59:59.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T16:59:59.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T16:59:59.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T16:59:59.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T16:59:59.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T16:59:59.432 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T16:59:59.432 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:59.432 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:59.432 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T16:59:59.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T16:59:59.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T16:59:59.592 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T16:59:59.592 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:59:59.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:59:59.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T16:59:59.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T16:59:59.593 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:59:59.593 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:59:59.594 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:59:59.594 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:59.594 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:59.594 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:59.594 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:59:59.595 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:59:59.595 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T16:59:59.685 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T16:59:59.685 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T16:59:59.685 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T16:59:59.686 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T16:59:59.686 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T16:59:59.686 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T16:59:59.686 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T16:59:59.686 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T16:59:59.687 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T16:59:59.687 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T16:59:59.687 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T16:59:59.687 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T16:59:59.687 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T16:59:59.688 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T16:59:59.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T16:59:59.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T16:59:59.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T16:59:59.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T16:59:59.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T16:59:59.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T16:59:59.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:00:00.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:00:00.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:00.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:00.651 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:00.651 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:00.652 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:00.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:00.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:00:00.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:00:00.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:00:00.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:00:00.957 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:00:00.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:00:01.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:00:01.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:00:01.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:00:01.969 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:00:01.969 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:01.969 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:01.969 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:00:01.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:00:01.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:00:02.049 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:00:02.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:00:02.058 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T16:59:59.463+0000 7fd1d078f6c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:00:02.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2020-09-08T17:00:02.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:450: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2020-09-08T17:00:02.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local objname=myobject 2020-09-08T17:00:02.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:453: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:00:02.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:02.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: setup_osds 7 2020-09-08T17:00:02.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=7 2020-09-08T17:00:02.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:00:02.061 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 7 - 1 2020-09-08T17:00:02.066 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 6 2020-09-08T17:00:02.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:00:02.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:00:02.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:00:02.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:00:02.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:00:02.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:00:02.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:00:02.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:02.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:00:02.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:00:02.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:00:02.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:00:02.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:00:02.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:00:02.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:00:02.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:00:02.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:00:02.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:02.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:02.071 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:02.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:02.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:02.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:02.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:02.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:00:02.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:00:02.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:00:02.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:00:02.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:00:02.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:00:02.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:00:02.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:02.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:00:02.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:00:02.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:00:02.076 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:00:02.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c0b6fb24-7582-4031-8d01-d71a9939743d 2020-09-08T17:00:02.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c0b6fb24-7582-4031-8d01-d71a9939743d' 2020-09-08T17:00:02.077 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:00:02.078 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 c0b6fb24-7582-4031-8d01-d71a9939743d 2020-09-08T17:00:02.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCSuFdfhwwOBRAAohOgFcQwM3YOR+XDicB6Qw== 2020-09-08T17:00:02.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCSuFdfhwwOBRAAohOgFcQwM3YOR+XDicB6Qw=="}' 2020-09-08T17:00:02.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c0b6fb24-7582-4031-8d01-d71a9939743d -i td/test-erasure-eio/0/new.json 2020-09-08T17:00:02.424 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:00:02.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:00:02.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCSuFdfhwwOBRAAohOgFcQwM3YOR+XDicB6Qw== --osd-uuid c0b6fb24-7582-4031-8d01-d71a9939743d 2020-09-08T17:00:02.459 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:02.464+0000 7f6f3c037ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:02.461 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:02.466+0000 7f6f3c037ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:02.463 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:02.468+0000 7f6f3c037ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:02.464 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:02.468+0000 7f6f3c037ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:00:02.464 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:02.469+0000 7f6f3c037ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:00:02.464 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:02.469+0000 7f6f3c037ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:00:02.510 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:02.514+0000 7f6f3c037ec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:00:07.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:00:07.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:00:07.156 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:00:07.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:00:07.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:00:07.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:00:07.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:00:07.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:07.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:00:07.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:00:07.655 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:00:07.676 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:07.681+0000 7ffa9c179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:07.678 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:07.682+0000 7ffa9c179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:07.678 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:07.683+0000 7ffa9c179ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:08.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:00:08.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:00:08.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:00:08.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:00:08.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:00:08.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:08.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:00:08.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:08.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:00:08.026 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:00:08.199 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:08.204+0000 7ffa9c179ec0 -1 Falling back to public interface 2020-09-08T17:00:08.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:09.047 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:09.051+0000 7ffa9c179ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:00:09.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:09.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:09.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:00:09.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:09.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:00:09.395 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:00:09.773 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/101808,v1:127.0.0.1:6803/101808] [v2:127.0.0.1:6804/101808,v1:127.0.0.1:6805/101808] exists,up c0b6fb24-7582-4031-8d01-d71a9939743d 2020-09-08T17:00:09.774 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 d6a699f7-91ef-44b7-873c-0e574d4cc67a 2020-09-08T17:00:09.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:00:09.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:00:09.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:00:09.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:00:09.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:00:09.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:00:09.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:00:09.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:00:09.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:00:09.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:00:09.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:09.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:00:09.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:00:09.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:00:09.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:00:09.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:00:09.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:00:09.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:00:09.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:00:09.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:00:09.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:09.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:09.779 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:09.779 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:09.779 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:09.780 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:09.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:09.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:00:09.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:00:09.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:00:09.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:00:09.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:00:09.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:00:09.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:00:09.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:09.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:00:09.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:00:09.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:00:09.783 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:00:09.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d6a699f7-91ef-44b7-873c-0e574d4cc67a 2020-09-08T17:00:09.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 d6a699f7-91ef-44b7-873c-0e574d4cc67a' 2020-09-08T17:00:09.783 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:00:09.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCZuFdfAL5ULxAAyEW6ai+pJvL371SaoDpHcA== 2020-09-08T17:00:09.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCZuFdfAL5ULxAAyEW6ai+pJvL371SaoDpHcA=="}' 2020-09-08T17:00:09.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d6a699f7-91ef-44b7-873c-0e574d4cc67a -i td/test-erasure-eio/1/new.json 2020-09-08T17:00:10.480 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:00:10.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:00:10.491 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCZuFdfAL5ULxAAyEW6ai+pJvL371SaoDpHcA== --osd-uuid d6a699f7-91ef-44b7-873c-0e574d4cc67a 2020-09-08T17:00:10.515 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:10.520+0000 7fa4c309dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:10.580 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:10.584+0000 7fa4c309dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:10.581 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:10.586+0000 7fa4c309dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:10.582 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:10.586+0000 7fa4c309dec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:00:10.582 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:10.586+0000 7fa4c309dec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:00:10.583 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:10.586+0000 7fa4c309dec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:00:10.647 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:10.652+0000 7fa4c309dec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:00:15.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:00:15.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:00:15.162 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:00:15.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:00:15.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:00:15.659 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:00:15.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:00:15.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:00:15.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:15.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:00:15.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:00:15.682 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:15.687+0000 7ff56c71eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:15.683 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:15.688+0000 7ff56c71eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:15.685 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:15.689+0000 7ff56c71eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:16.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:00:16.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:00:16.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:00:16.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:00:16.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:00:16.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:16.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:00:16.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:16.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:00:16.050 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:00:16.201 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:16.205+0000 7ff56c71eec0 -1 Falling back to public interface 2020-09-08T17:00:16.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:17.145 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:17.149+0000 7ff56c71eec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:00:17.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:17.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:17.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:00:17.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:17.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:00:17.418 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:00:17.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:18.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:18.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:18.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:00:18.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:18.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:00:18.799 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:00:19.191 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/102802,v1:127.0.0.1:6811/102802] [v2:127.0.0.1:6812/102802,v1:127.0.0.1:6813/102802] exists,up d6a699f7-91ef-44b7-873c-0e574d4cc67a 2020-09-08T17:00:19.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:00:19.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:00:19.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:00:19.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:00:19.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:00:19.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:00:19.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:00:19.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:00:19.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:00:19.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:00:19.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:19.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:00:19.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:00:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:00:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:00:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:00:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:00:19.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:00:19.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:00:19.196 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:00:19.196 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:19.196 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:19.196 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:19.197 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:19.197 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:19.197 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:19.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:19.197 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:00:19.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:00:19.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:00:19.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:00:19.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:00:19.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:00:19.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:00:19.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:19.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:00:19.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:00:19.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:00:19.199 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:00:19.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c1f555a1-23cb-41e3-90e9-86b31f932185 2020-09-08T17:00:19.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 c1f555a1-23cb-41e3-90e9-86b31f932185' 2020-09-08T17:00:19.201 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:00:19.201 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 c1f555a1-23cb-41e3-90e9-86b31f932185 2020-09-08T17:00:19.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCjuFdfne+uDBAA5biIKjRaOoyMjN89kMoLCg== 2020-09-08T17:00:19.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCjuFdfne+uDBAA5biIKjRaOoyMjN89kMoLCg=="}' 2020-09-08T17:00:19.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c1f555a1-23cb-41e3-90e9-86b31f932185 -i td/test-erasure-eio/2/new.json 2020-09-08T17:00:19.690 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:00:19.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:00:19.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCjuFdfne+uDBAA5biIKjRaOoyMjN89kMoLCg== --osd-uuid c1f555a1-23cb-41e3-90e9-86b31f932185 2020-09-08T17:00:19.726 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:19.731+0000 7f6d93c13ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:19.727 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:19.733+0000 7f6d93c13ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:19.728 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:19.734+0000 7f6d93c13ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:19.729 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:19.735+0000 7f6d93c13ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:00:19.730 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:19.735+0000 7f6d93c13ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:00:19.730 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:19.735+0000 7f6d93c13ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:00:19.831 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:19.836+0000 7f6d93c13ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:00:24.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:00:24.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:00:24.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:00:24.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:00:24.668 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:00:25.160 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:00:25.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:00:25.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:25.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:00:25.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:00:25.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:00:25.191 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:25.188+0000 7f98fbc4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:25.191 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:25.190+0000 7f98fbc4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:25.191 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:25.192+0000 7f98fbc4cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:25.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:00:25.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:00:25.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:00:25.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:00:25.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:00:25.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:25.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:00:25.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:25.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:00:25.559 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:00:25.702 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:25.708+0000 7f98fbc4cec0 -1 Falling back to public interface 2020-09-08T17:00:25.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:26.569 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:26.575+0000 7f98fbc4cec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:00:26.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:26.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:26.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:00:26.930 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:00:26.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:00:26.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:27.373 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/103822,v1:127.0.0.1:6819/103822] [v2:127.0.0.1:6820/103822,v1:127.0.0.1:6821/103822] exists,up c1f555a1-23cb-41e3-90e9-86b31f932185 2020-09-08T17:00:27.373 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 a0823f7c-2aee-4268-8fae-e188bde142cf 2020-09-08T17:00:27.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:00:27.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:00:27.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:00:27.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:00:27.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:00:27.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:00:27.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:00:27.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:00:27.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:00:27.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:00:27.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:27.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:00:27.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:00:27.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:00:27.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:00:27.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:00:27.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:00:27.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:00:27.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:00:27.378 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:00:27.379 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:27.379 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:27.379 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:27.379 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:27.380 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:27.380 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:27.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:27.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:00:27.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:00:27.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:00:27.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:00:27.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:00:27.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:00:27.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:00:27.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:27.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:00:27.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:00:27.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:00:27.383 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:00:27.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a0823f7c-2aee-4268-8fae-e188bde142cf 2020-09-08T17:00:27.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 a0823f7c-2aee-4268-8fae-e188bde142cf' 2020-09-08T17:00:27.383 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:00:27.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCruFdf7BlsFxAAiaWDpFIPhPqCIEjlOX31ng== 2020-09-08T17:00:27.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCruFdf7BlsFxAAiaWDpFIPhPqCIEjlOX31ng=="}' 2020-09-08T17:00:27.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a0823f7c-2aee-4268-8fae-e188bde142cf -i td/test-erasure-eio/3/new.json 2020-09-08T17:00:28.226 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:00:28.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:00:28.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCruFdf7BlsFxAAiaWDpFIPhPqCIEjlOX31ng== --osd-uuid a0823f7c-2aee-4268-8fae-e188bde142cf 2020-09-08T17:00:28.262 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:28.267+0000 7f64a876bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:28.307 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:28.313+0000 7f64a876bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:28.308 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:28.314+0000 7f64a876bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:28.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:28.315+0000 7f64a876bec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:00:28.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:28.315+0000 7f64a876bec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:00:28.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:28.315+0000 7f64a876bec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:00:28.474 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:28.480+0000 7f64a876bec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:00:33.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:00:33.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:00:33.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:00:33.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:00:33.487 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:00:33.984 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:00:33.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:00:33.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:33.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:00:33.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:00:33.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:00:34.007 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:34.011+0000 7fa101ed9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:34.008 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:34.013+0000 7fa101ed9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:34.010 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:34.014+0000 7fa101ed9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:34.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:00:34.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:00:34.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:00:34.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:00:34.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:00:34.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:34.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:00:34.385 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:34.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:00:34.386 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:00:34.526 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:34.530+0000 7fa101ed9ec0 -1 Falling back to public interface 2020-09-08T17:00:34.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:35.341 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:35.346+0000 7fa101ed9ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:00:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:35.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:35.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:00:35.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:35.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:00:35.823 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:00:36.289 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/104816,v1:127.0.0.1:6827/104816] [v2:127.0.0.1:6828/104816,v1:127.0.0.1:6829/104816] exists,up a0823f7c-2aee-4268-8fae-e188bde142cf 2020-09-08T17:00:36.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:00:36.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:00:36.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:00:36.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:00:36.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 4 2020-09-08T17:00:36.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:00:36.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:00:36.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T17:00:36.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:00:36.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/4 2020-09-08T17:00:36.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:36.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:00:36.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:00:36.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:00:36.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2020-09-08T17:00:36.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2020-09-08T17:00:36.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:00:36.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:00:36.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:00:36.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:00:36.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:36.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:36.294 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:36.294 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:36.294 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:36.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:36.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:36.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:00:36.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:00:36.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:00:36.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:00:36.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:00:36.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:00:36.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:00:36.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:36.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:00:36.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:00:36.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/4 2020-09-08T17:00:36.297 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:00:36.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0311eaee-2491-4aaa-8fec-44302dd8c0bd 2020-09-08T17:00:36.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 0311eaee-2491-4aaa-8fec-44302dd8c0bd' 2020-09-08T17:00:36.298 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:00:36.299 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 0311eaee-2491-4aaa-8fec-44302dd8c0bd 2020-09-08T17:00:36.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC0uFdfXop+EhAAEC4tPtK3/NoAEdZLbqNaCg== 2020-09-08T17:00:36.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC0uFdfXop+EhAAEC4tPtK3/NoAEdZLbqNaCg=="}' 2020-09-08T17:00:36.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0311eaee-2491-4aaa-8fec-44302dd8c0bd -i td/test-erasure-eio/4/new.json 2020-09-08T17:00:37.047 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T17:00:37.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/4/new.json 2020-09-08T17:00:37.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC0uFdfXop+EhAAEC4tPtK3/NoAEdZLbqNaCg== --osd-uuid 0311eaee-2491-4aaa-8fec-44302dd8c0bd 2020-09-08T17:00:37.083 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:37.087+0000 7efe377f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:37.084 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:37.089+0000 7efe377f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:37.086 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:37.091+0000 7efe377f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:37.087 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:37.091+0000 7efe377f9ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:00:37.087 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:37.091+0000 7efe377f9ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:00:37.087 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:37.091+0000 7efe377f9ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:00:37.231 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:37.236+0000 7efe377f9ec0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2020-09-08T17:00:42.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2020-09-08T17:00:42.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:00:42.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T17:00:42.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:00:42.093 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T17:00:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T17:00:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:00:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:00:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:00:42.595 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T17:00:42.616 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:42.621+0000 7f84d1033ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:42.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:42.623+0000 7f84d1033ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:42.620 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:42.624+0000 7f84d1033ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:42.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T17:00:42.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:00:42.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T17:00:42.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:00:42.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:00:42.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:42.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:00:42.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:42.981 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:00:42.981 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:00:43.136 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:43.141+0000 7f84d1033ec0 -1 Falling back to public interface 2020-09-08T17:00:43.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:43.942 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:43.947+0000 7f84d1033ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T17:00:44.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:44.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:44.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:00:44.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:44.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:00:44.351 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:00:44.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:45.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:45.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:45.744 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:00:45.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:00:45.745 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:45.745 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:00:46.178 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 31 up_thru 31 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/105818,v1:127.0.0.1:6835/105818] [v2:127.0.0.1:6836/105818,v1:127.0.0.1:6837/105818] exists,up 0311eaee-2491-4aaa-8fec-44302dd8c0bd 2020-09-08T17:00:46.178 INFO:tasks.workunit.client.0.smithi120.stdout:add osd5 cc3f6887-6549-40d5-857f-2463b4108c63 2020-09-08T17:00:46.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:00:46.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:00:46.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:00:46.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:00:46.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 5 2020-09-08T17:00:46.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:00:46.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:00:46.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-09-08T17:00:46.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:00:46.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/5 2020-09-08T17:00:46.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:46.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:00:46.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:00:46.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:00:46.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2020-09-08T17:00:46.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2020-09-08T17:00:46.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:00:46.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:00:46.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:00:46.183 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:00:46.183 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:46.183 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:46.184 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:46.184 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:46.184 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:46.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:46.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:00:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:00:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:00:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:00:46.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:00:46.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:00:46.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:00:46.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:46.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:00:46.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:00:46.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/5 2020-09-08T17:00:46.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:00:46.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cc3f6887-6549-40d5-857f-2463b4108c63 2020-09-08T17:00:46.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 cc3f6887-6549-40d5-857f-2463b4108c63' 2020-09-08T17:00:46.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:00:46.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC+uFdf6IbcCxAACrjN/lQK/BggjuwyTiqdvQ== 2020-09-08T17:00:46.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC+uFdf6IbcCxAACrjN/lQK/BggjuwyTiqdvQ=="}' 2020-09-08T17:00:46.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cc3f6887-6549-40d5-857f-2463b4108c63 -i td/test-erasure-eio/5/new.json 2020-09-08T17:00:46.788 INFO:tasks.workunit.client.0.smithi120.stdout:5 2020-09-08T17:00:46.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/5/new.json 2020-09-08T17:00:46.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC+uFdf6IbcCxAACrjN/lQK/BggjuwyTiqdvQ== --osd-uuid cc3f6887-6549-40d5-857f-2463b4108c63 2020-09-08T17:00:46.823 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:46.828+0000 7fa104a85ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:46.825 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:46.830+0000 7fa104a85ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:46.827 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:46.832+0000 7fa104a85ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:46.828 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:46.832+0000 7fa104a85ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:00:46.828 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:46.832+0000 7fa104a85ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:00:46.828 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:46.832+0000 7fa104a85ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:00:47.045 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:47.050+0000 7fa104a85ec0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2020-09-08T17:00:51.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2020-09-08T17:00:51.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:00:51.565 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd5 key to auth repository 2020-09-08T17:00:51.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-09-08T17:00:51.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:00:52.065 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.5 2020-09-08T17:00:52.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-09-08T17:00:52.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:52.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:00:52.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:00:52.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:00:52.088 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:52.092+0000 7f9f3fe11ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:52.090 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:52.094+0000 7f9f3fe11ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:52.090 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:52.095+0000 7f9f3fe11ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:52.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-09-08T17:00:52.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:00:52.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-09-08T17:00:52.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:00:52.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:00:52.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:52.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:00:52.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:52.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:00:52.466 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:00:52.606 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:52.610+0000 7f9f3fe11ec0 -1 Falling back to public interface 2020-09-08T17:00:52.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:53.398 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:53.402+0000 7f9f3fe11ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-09-08T17:00:53.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:53.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:53.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:00:53.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:53.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:00:53.839 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:00:54.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:00:55.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:00:55.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:00:55.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:00:55.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:00:55.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:00:55.307 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:00:55.687 INFO:tasks.workunit.client.0.smithi120.stdout:osd.5 up in weight 1 up_from 38 up_thru 38 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/106838,v1:127.0.0.1:6843/106838] [v2:127.0.0.1:6844/106838,v1:127.0.0.1:6845/106838] exists,up cc3f6887-6549-40d5-857f-2463b4108c63 2020-09-08T17:00:55.688 INFO:tasks.workunit.client.0.smithi120.stdout:add osd6 e2cbabbb-b4c1-4346-81b3-f83f463ed92c 2020-09-08T17:00:55.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:00:55.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:00:55.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:00:55.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:00:55.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 6 2020-09-08T17:00:55.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:00:55.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:00:55.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2020-09-08T17:00:55.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:00:55.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/6 2020-09-08T17:00:55.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:00:55.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:00:55.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:00:55.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:00:55.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2020-09-08T17:00:55.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2020-09-08T17:00:55.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:00:55.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:00:55.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:00:55.692 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:00:55.692 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:00:55.692 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:00:55.692 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:00:55.693 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:00:55.693 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:00:55.693 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:55.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:00:55.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:00:55.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:00:55.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:00:55.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:00:55.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:00:55.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:00:55.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:00:55.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:00:55.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:00:55.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:00:55.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/6 2020-09-08T17:00:55.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:00:55.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e2cbabbb-b4c1-4346-81b3-f83f463ed92c 2020-09-08T17:00:55.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 e2cbabbb-b4c1-4346-81b3-f83f463ed92c' 2020-09-08T17:00:55.696 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:00:55.702 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDHuFdfpIA2KhAAOJzhYtJ4PDfXicCXLJfhpw== 2020-09-08T17:00:55.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDHuFdfpIA2KhAAOJzhYtJ4PDfXicCXLJfhpw=="}' 2020-09-08T17:00:55.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e2cbabbb-b4c1-4346-81b3-f83f463ed92c -i td/test-erasure-eio/6/new.json 2020-09-08T17:00:56.181 INFO:tasks.workunit.client.0.smithi120.stdout:6 2020-09-08T17:00:56.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/6/new.json 2020-09-08T17:00:56.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDHuFdfpIA2KhAAOJzhYtJ4PDfXicCXLJfhpw== --osd-uuid e2cbabbb-b4c1-4346-81b3-f83f463ed92c 2020-09-08T17:00:56.216 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:56.220+0000 7f7dc6d05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:56.217 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:56.222+0000 7f7dc6d05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:56.218 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:56.223+0000 7f7dc6d05ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:00:56.219 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:56.223+0000 7f7dc6d05ec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:00:56.219 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:56.223+0000 7f7dc6d05ec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:00:56.219 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:56.223+0000 7f7dc6d05ec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:00:56.291 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:00:56.296+0000 7f7dc6d05ec0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2020-09-08T17:01:01.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2020-09-08T17:01:01.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:01:01.225 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2020-09-08T17:01:01.225 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:01:01.226 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd6 key to auth repository 2020-09-08T17:01:01.729 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.6 2020-09-08T17:01:01.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2020-09-08T17:01:01.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:01.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:01:01.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:01:01.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:01:01.752 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:01.756+0000 7fb607966ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:01.753 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:01.758+0000 7fb607966ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:01.755 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:01.760+0000 7fb607966ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:02.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2020-09-08T17:01:02.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:01:02.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-09-08T17:01:02.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:01:02.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:01:02.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:02.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:01:02.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:02.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:01:02.106 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:01:02.272 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:02.276+0000 7fb607966ec0 -1 Falling back to public interface 2020-09-08T17:01:02.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:01:03.139 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:03.144+0000 7fb607966ec0 -1 osd.6 0 log_to_monitors {default=true} 2020-09-08T17:01:03.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:01:03.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:03.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:01:03.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:03.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:01:03.504 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:01:03.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:01:04.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:01:04.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:04.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:01:04.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:04.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:01:04.950 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:01:05.422 INFO:tasks.workunit.client.0.smithi120.stdout:osd.6 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/107871,v1:127.0.0.1:6851/107871] [v2:127.0.0.1:6852/107871,v1:127.0.0.1:6853/107871] exists,up e2cbabbb-b4c1-4346-81b3-f83f463ed92c 2020-09-08T17:01:05.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:01:05.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:01:05.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:01:05.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:01:05.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:01:05.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:01:05.424 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:01:05.424 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:05.425 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:05.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:01:05.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:01:05.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:01:05.513 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:01:05.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:01:05.524 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:00:08.210+0000 7ffa9c179ec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:01:05.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:01:05.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:458: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2020-09-08T17:01:05.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2020-09-08T17:01:05.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:01:05.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:01:05.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=3 2020-09-08T17:01:05.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:01:05.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=2 2020-09-08T17:01:05.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:01:05.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2020-09-08T17:01:06.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:01:06.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:01:06.950 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:01:06.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:01:07.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:01:07.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:01:07.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:01:07.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:01:07.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:01:07.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:01:07.962 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:01:07.963 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:01:07.963 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:01:07.963 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:01:07.963 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:01:08.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:01:08.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:01:08.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:01:08.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:01:08.066 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:01:08.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:01:08.479 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:01:08.479 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:01:08.480 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:01:08.480 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:01:08.480 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:01:08.480 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:01:08.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:01:08.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:08.481 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:01:08.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803790 2020-09-08T17:01:08.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803790 2020-09-08T17:01:08.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790' 2020-09-08T17:01:08.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:08.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:01:08.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574861 2020-09-08T17:01:08.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574861 2020-09-08T17:01:08.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861' 2020-09-08T17:01:08.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:08.755 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:01:08.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378635 2020-09-08T17:01:08.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378635 2020-09-08T17:01:08.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635' 2020-09-08T17:01:08.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:08.875 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:01:08.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149705 2020-09-08T17:01:08.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149705 2020-09-08T17:01:08.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705' 2020-09-08T17:01:08.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:08.996 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:01:09.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=133143986183 2020-09-08T17:01:09.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 133143986183 2020-09-08T17:01:09.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705 4-133143986183' 2020-09-08T17:01:09.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:09.116 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T17:01:09.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757253 2020-09-08T17:01:09.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757253 2020-09-08T17:01:09.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705 4-133143986183 5-163208757253' 2020-09-08T17:01:09.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:09.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T17:01:09.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528323 2020-09-08T17:01:09.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528323 2020-09-08T17:01:09.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705 4-133143986183 5-163208757253 6-193273528323' 2020-09-08T17:01:09.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:09.359 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803790 2020-09-08T17:01:09.359 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:09.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:01:09.361 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803790 2020-09-08T17:01:09.361 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:09.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803790 2020-09-08T17:01:09.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803790' 2020-09-08T17:01:09.361 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803790 2020-09-08T17:01:09.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:01:09.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803790 2020-09-08T17:01:09.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:01:10.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:01:10.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:01:11.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803790 -lt 25769803790 2020-09-08T17:01:11.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:11.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574861 2020-09-08T17:01:11.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:11.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:01:11.096 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574861 2020-09-08T17:01:11.096 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:11.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574861 2020-09-08T17:01:11.097 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574861' 2020-09-08T17:01:11.097 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:01:11.098 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574861 2020-09-08T17:01:11.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574861 -lt 55834574861 2020-09-08T17:01:11.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:11.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378635 2020-09-08T17:01:11.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:11.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:01:11.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378635 2020-09-08T17:01:11.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:11.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378635 2020-09-08T17:01:11.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378635' 2020-09-08T17:01:11.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:01:11.504 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378635 2020-09-08T17:01:11.872 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378635 -lt 81604378635 2020-09-08T17:01:11.872 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:11.873 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149705 2020-09-08T17:01:11.873 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:11.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:01:11.874 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149705 2020-09-08T17:01:11.875 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:11.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149705 2020-09-08T17:01:11.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149705' 2020-09-08T17:01:11.876 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:01:11.876 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149705 2020-09-08T17:01:12.260 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149705 -lt 111669149705 2020-09-08T17:01:12.261 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:12.261 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-133143986183 2020-09-08T17:01:12.262 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:12.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:01:12.263 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-133143986183 2020-09-08T17:01:12.263 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:12.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=133143986183 2020-09-08T17:01:12.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 133143986183' 2020-09-08T17:01:12.264 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:01:12.264 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 133143986183 2020-09-08T17:01:12.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 133143986184 -lt 133143986183 2020-09-08T17:01:12.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:12.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-163208757253 2020-09-08T17:01:12.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:12.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T17:01:12.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-163208757253 2020-09-08T17:01:12.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:12.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757253 2020-09-08T17:01:12.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 163208757253' 2020-09-08T17:01:12.632 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T17:01:12.633 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 163208757253 2020-09-08T17:01:12.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757254 -lt 163208757253 2020-09-08T17:01:12.998 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:12.998 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-193273528323 2020-09-08T17:01:12.999 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:12.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T17:01:12.999 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-193273528323 2020-09-08T17:01:13.000 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:13.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528323 2020-09-08T17:01:13.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 193273528323' 2020-09-08T17:01:13.001 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T17:01:13.001 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 193273528323 2020-09-08T17:01:13.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528324 -lt 193273528323 2020-09-08T17:01:13.392 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:01:13.392 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:01:13.393 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:01:13.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:01:13.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:01:13.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:01:13.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:01:13.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:01:13.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:01:13.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:01:13.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:01:14.248 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:01:14.248 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:01:14.249 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:01:14.249 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:01:14.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:01:14.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:01:14.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:01:14.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:461: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2020-09-08T17:01:14.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:01:14.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:01:14.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=test1 2020-09-08T17:01:14.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:01:14.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:01:14.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:01:14.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:01:14.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2020-09-08T17:01:14.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2020-09-08T17:01:14.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:01:14.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:01:14.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=test2 2020-09-08T17:01:14.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:01:14.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:01:14.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:01:14.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:14.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:01:14.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2020-09-08T17:01:15.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2020-09-08T17:01:15.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:01:15.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:01:15.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=test3 2020-09-08T17:01:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:01:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:01:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:15.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:01:15.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:01:15.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:01:15.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2020-09-08T17:01:15.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:465: TEST_ec_recovery_multiple_objects: ceph osd out 0 2020-09-08T17:01:16.124 INFO:tasks.workunit.client.0.smithi120.stderr:osd.0 is already out. 2020-09-08T17:01:16.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:468: TEST_ec_recovery_multiple_objects: wait_for_clean 2020-09-08T17:01:16.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:01:16.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:01:16.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:01:16.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:01:16.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:01:16.134 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:01:16.134 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:01:16.135 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:01:16.135 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:01:16.135 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:01:16.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:01:16.224 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:01:16.225 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:01:16.225 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:01:16.225 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:01:16.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:01:16.645 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:01:16.645 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:01:16.645 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:01:16.645 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:01:16.645 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:01:16.646 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:01:16.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:01:16.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:16.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:01:16.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803794 2020-09-08T17:01:16.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803794 2020-09-08T17:01:16.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794' 2020-09-08T17:01:16.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:16.787 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:01:16.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574864 2020-09-08T17:01:16.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574864 2020-09-08T17:01:16.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574864' 2020-09-08T17:01:16.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:16.969 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:01:17.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378638 2020-09-08T17:01:17.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378638 2020-09-08T17:01:17.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574864 2-81604378638' 2020-09-08T17:01:17.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:17.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:01:17.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149709 2020-09-08T17:01:17.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149709 2020-09-08T17:01:17.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574864 2-81604378638 3-111669149709' 2020-09-08T17:01:17.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:17.212 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:01:17.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=133143986187 2020-09-08T17:01:17.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 133143986187 2020-09-08T17:01:17.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574864 2-81604378638 3-111669149709 4-133143986187' 2020-09-08T17:01:17.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:17.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T17:01:17.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757257 2020-09-08T17:01:17.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757257 2020-09-08T17:01:17.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574864 2-81604378638 3-111669149709 4-133143986187 5-163208757257' 2020-09-08T17:01:17.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:01:17.463 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T17:01:17.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528327 2020-09-08T17:01:17.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528327 2020-09-08T17:01:17.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574864 2-81604378638 3-111669149709 4-133143986187 5-163208757257 6-193273528327' 2020-09-08T17:01:17.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:17.576 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803794 2020-09-08T17:01:17.576 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:17.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:01:17.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803794 2020-09-08T17:01:17.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:17.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803794 2020-09-08T17:01:17.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803794' 2020-09-08T17:01:17.579 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803794 2020-09-08T17:01:17.579 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:01:17.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 0 -lt 25769803794 2020-09-08T17:01:17.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:01:18.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:01:18.952 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:01:19.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803794 -lt 25769803794 2020-09-08T17:01:19.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:19.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574864 2020-09-08T17:01:19.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:19.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:01:19.389 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574864 2020-09-08T17:01:19.390 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:19.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574864 2020-09-08T17:01:19.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574864' 2020-09-08T17:01:19.391 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:01:19.391 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574864 2020-09-08T17:01:19.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574864 -lt 55834574864 2020-09-08T17:01:19.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:19.752 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378638 2020-09-08T17:01:19.753 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:19.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:01:19.754 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378638 2020-09-08T17:01:19.754 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:19.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378638 2020-09-08T17:01:19.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378638' 2020-09-08T17:01:19.755 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:01:19.755 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378638 2020-09-08T17:01:20.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378639 -lt 81604378638 2020-09-08T17:01:20.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:20.197 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149709 2020-09-08T17:01:20.197 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:20.198 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:01:20.198 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149709 2020-09-08T17:01:20.198 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:20.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149709 2020-09-08T17:01:20.199 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149709' 2020-09-08T17:01:20.200 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:01:20.200 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149709 2020-09-08T17:01:20.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149709 -lt 111669149709 2020-09-08T17:01:20.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:20.573 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-133143986187 2020-09-08T17:01:20.573 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:20.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:01:20.575 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-133143986187 2020-09-08T17:01:20.575 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:20.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=133143986187 2020-09-08T17:01:20.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 133143986187' 2020-09-08T17:01:20.575 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:01:20.576 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 133143986187 2020-09-08T17:01:20.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 133143986187 -lt 133143986187 2020-09-08T17:01:20.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:20.938 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-163208757257 2020-09-08T17:01:20.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:20.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T17:01:20.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-163208757257 2020-09-08T17:01:20.940 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:20.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757257 2020-09-08T17:01:20.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 163208757257' 2020-09-08T17:01:20.941 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T17:01:20.941 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 163208757257 2020-09-08T17:01:21.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757258 -lt 163208757257 2020-09-08T17:01:21.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:01:21.303 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-193273528327 2020-09-08T17:01:21.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:01:21.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T17:01:21.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-193273528327 2020-09-08T17:01:21.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:01:21.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528327 2020-09-08T17:01:21.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 193273528327' 2020-09-08T17:01:21.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T17:01:21.306 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 193273528327 2020-09-08T17:01:21.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528328 -lt 193273528327 2020-09-08T17:01:21.730 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:01:21.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:01:21.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:01:22.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:01:22.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:01:22.214 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:01:22.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:01:22.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:01:22.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:01:22.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:01:22.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:01:22.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:01:22.569 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:01:22.570 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:01:22.570 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:01:23.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:01:23.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:01:23.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:01:23.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:470: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2020-09-08T17:01:23.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:01:23.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:01:23.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=test1 2020-09-08T17:01:23.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:01:23.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:01:23.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2020-09-08T17:01:23.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:01:23.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:01:23.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2020-09-08T17:01:23.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:01:23.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:01:23.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=test2 2020-09-08T17:01:23.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:01:23.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:01:23.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2020-09-08T17:01:23.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:01:23.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:01:23.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2020-09-08T17:01:23.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:01:23.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:01:23.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=test3 2020-09-08T17:01:23.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:01:23.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:01:23.247 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2020-09-08T17:01:23.270 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:01:23.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:01:23.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:474: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:01:23.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:01:23.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:01:23.768 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:01:23.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:01:24.305 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:01:24.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:01:24.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:01:24.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:01:24.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:01:24.314 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:01:24.314 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:01:24.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:01:24.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:01:24.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:01:24.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:01:24.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:01:24.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:01:24.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:01:24.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:01:24.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:01:24.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:01:24.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:01:24.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:01:24.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:01:24.428 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:01:24.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:01:24.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:01:24.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:01:24.438 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:01:24.438 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:24.438 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:24.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:01:24.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:01:24.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:01:24.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:01:24.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:01:24.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:01:24.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:01:24.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:01:24.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:01:24.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:01:24.442 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:01:24.442 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:01:24.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:01:24.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:01:24.443 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:01:24.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:01:24.444 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:01:24.444 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:01:24.444 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:01:24.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:01:24.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:01:24.445 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:01:24.446 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:01:24.446 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:01:24.446 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:01:24.447 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:01:24.447 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:01:24.447 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:01:24.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:01:24.449 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:01:24.449 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:24.449 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:24.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:01:24.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:01:24.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:01:24.450 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:01:24.451 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:01:24.451 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:24.451 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:24.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:01:24.453 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:01:24.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:01:24.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:01:24.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:01:24.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:01:24.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:01:24.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:01:24.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:01:24.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:01:24.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:01:24.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:01:24.794 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:01:24.795 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:01:24.795 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:01:24.795 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:01:24.796 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:24.796 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:24.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:24.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:01:24.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:01:24.981 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:01:24.981 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:01:24.981 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:01:24.981 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:01:24.982 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:01:24.982 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:01:24.982 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:01:24.982 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:01:24.982 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:01:24.983 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:24.983 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:24.983 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:01:24.984 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:01:24.984 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:01:25.069 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:01:25.070 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:01:25.070 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:01:25.070 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:01:25.070 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:01:25.070 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:01:25.071 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:01:25.071 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:01:25.071 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:01:25.071 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:25.071 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:25.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:01:25.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:01:25.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:01:25.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:01:25.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:01:25.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:01:25.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:01:25.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:01:25.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:01:25.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:01:26.038 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:01:26.039 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:01:26.039 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:01:26.039 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:01:26.039 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:26.040 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:26.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:26.040 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:01:26.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:01:26.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:01:26.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:01:26.356 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:01:26.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:01:27.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:01:27.367 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:01:27.367 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:01:27.367 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:01:27.367 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:27.368 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:27.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:01:27.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:01:27.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:01:27.448 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:01:27.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:01:27.459 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:01:24.827+0000 7f21cd1f16c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:01:27.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2020-09-08T17:01:27.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:479: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2020-09-08T17:01:27.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2020-09-08T17:01:27.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:482: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:01:27.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:01:27.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2020-09-08T17:01:27.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=7 2020-09-08T17:01:27.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:01:27.461 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 7 - 1 2020-09-08T17:01:27.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 6 2020-09-08T17:01:27.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:01:27.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:01:27.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:01:27.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:01:27.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:01:27.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:01:27.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:01:27.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:01:27.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:01:27.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:01:27.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:01:27.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:01:27.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:01:27.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:01:27.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:01:27.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:01:27.468 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:01:27.468 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:01:27.468 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:01:27.468 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:01:27.468 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:27.469 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:27.469 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:27.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:27.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:01:27.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:01:27.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:01:27.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:01:27.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:01:27.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:01:27.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:01:27.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:27.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:01:27.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:01:27.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:01:27.472 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:01:27.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0ef2ec9d-e8e2-40a1-b3f8-c7d5e1e20ca7 2020-09-08T17:01:27.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 0ef2ec9d-e8e2-40a1-b3f8-c7d5e1e20ca7' 2020-09-08T17:01:27.472 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:01:27.472 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 0ef2ec9d-e8e2-40a1-b3f8-c7d5e1e20ca7 2020-09-08T17:01:27.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDnuFdfkTjqHBAAOCzZBj4C+B+v1TnDWjw1lg== 2020-09-08T17:01:27.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDnuFdfkTjqHBAAOCzZBj4C+B+v1TnDWjw1lg=="}' 2020-09-08T17:01:27.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0ef2ec9d-e8e2-40a1-b3f8-c7d5e1e20ca7 -i td/test-erasure-eio/0/new.json 2020-09-08T17:01:27.855 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:01:27.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:01:27.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDnuFdfkTjqHBAAOCzZBj4C+B+v1TnDWjw1lg== --osd-uuid 0ef2ec9d-e8e2-40a1-b3f8-c7d5e1e20ca7 2020-09-08T17:01:27.890 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:27.895+0000 7f705d2c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:27.892 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:27.897+0000 7f705d2c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:27.894 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:27.899+0000 7f705d2c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:27.910 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:27.899+0000 7f705d2c9ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:01:27.910 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:27.899+0000 7f705d2c9ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:01:27.911 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:27.899+0000 7f705d2c9ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:01:28.001 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:28.005+0000 7f705d2c9ec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:01:32.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:01:32.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:01:32.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:01:32.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:01:32.574 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:01:33.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:01:33.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:33.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:01:33.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:01:33.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:01:33.066 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:01:33.088 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:33.092+0000 7f03206faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:33.089 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:33.094+0000 7f03206faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:33.090 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:33.095+0000 7f03206faec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:33.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:01:33.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:01:33.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:01:33.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:01:33.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:01:33.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:33.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:01:33.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:33.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:01:33.438 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:01:33.607 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:33.612+0000 7f03206faec0 -1 Falling back to public interface 2020-09-08T17:01:33.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:01:34.455 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:34.460+0000 7f03206faec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:01:34.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:01:34.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:34.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:01:34.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:34.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:01:34.824 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:01:35.202 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/110692,v1:127.0.0.1:6803/110692] [v2:127.0.0.1:6804/110692,v1:127.0.0.1:6805/110692] exists,up 0ef2ec9d-e8e2-40a1-b3f8-c7d5e1e20ca7 2020-09-08T17:01:35.202 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 046440b9-4ebf-4205-bb25-9589694d8489 2020-09-08T17:01:35.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:01:35.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:01:35.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:01:35.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:01:35.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:01:35.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:01:35.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:01:35.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:01:35.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:01:35.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:01:35.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:01:35.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:01:35.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:01:35.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:01:35.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:01:35.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:01:35.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:01:35.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:01:35.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:01:35.207 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:01:35.207 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:01:35.207 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:01:35.207 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:01:35.208 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:35.208 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:35.208 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:35.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:35.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:01:35.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:01:35.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:01:35.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:01:35.209 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:01:35.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:01:35.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:01:35.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:35.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:01:35.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:01:35.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:01:35.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:01:35.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=046440b9-4ebf-4205-bb25-9589694d8489 2020-09-08T17:01:35.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 046440b9-4ebf-4205-bb25-9589694d8489' 2020-09-08T17:01:35.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:01:35.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDvuFdfEQxDDRAA8xqdzemqA0jd9HWSf+Z1SA== 2020-09-08T17:01:35.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDvuFdfEQxDDRAA8xqdzemqA0jd9HWSf+Z1SA=="}' 2020-09-08T17:01:35.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 046440b9-4ebf-4205-bb25-9589694d8489 -i td/test-erasure-eio/1/new.json 2020-09-08T17:01:36.274 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:01:36.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:01:36.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDvuFdfEQxDDRAA8xqdzemqA0jd9HWSf+Z1SA== --osd-uuid 046440b9-4ebf-4205-bb25-9589694d8489 2020-09-08T17:01:36.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:36.314+0000 7f7491de5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:36.332 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:36.337+0000 7f7491de5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:36.333 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:36.338+0000 7f7491de5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:36.334 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:36.338+0000 7f7491de5ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:01:36.334 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:36.338+0000 7f7491de5ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:01:36.334 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:36.338+0000 7f7491de5ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:01:36.399 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:36.403+0000 7f7491de5ec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:01:41.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:01:41.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:01:41.100 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:01:41.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:01:41.101 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:01:41.597 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:01:41.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:01:41.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:41.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:01:41.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:01:41.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:01:41.619 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:41.624+0000 7f35eb66bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:41.621 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:41.626+0000 7f35eb66bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:41.623 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:41.628+0000 7f35eb66bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:41.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:01:41.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:01:41.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:01:41.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:01:41.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:01:41.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:41.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:01:41.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:41.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:01:41.970 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:01:42.139 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:42.143+0000 7f35eb66bec0 -1 Falling back to public interface 2020-09-08T17:01:42.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:01:42.991 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:42.995+0000 7f35eb66bec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:01:43.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:01:43.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:43.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:01:43.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:01:43.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:43.410 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:01:43.786 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/111694,v1:127.0.0.1:6811/111694] [v2:127.0.0.1:6812/111694,v1:127.0.0.1:6813/111694] exists,up 046440b9-4ebf-4205-bb25-9589694d8489 2020-09-08T17:01:43.787 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 8c36d4ee-fc94-4a61-8cde-343483e43de6 2020-09-08T17:01:43.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:01:43.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:01:43.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:01:43.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:01:43.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:01:43.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:01:43.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:01:43.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:01:43.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:01:43.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:01:43.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:01:43.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:01:43.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:01:43.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:01:43.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:01:43.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:01:43.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:01:43.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:01:43.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:01:43.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:01:43.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:01:43.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:01:43.792 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:01:43.792 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:43.792 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:43.792 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:43.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:43.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:01:43.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:01:43.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:01:43.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:01:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:01:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:01:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:01:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:01:43.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:01:43.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:01:43.795 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:01:43.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8c36d4ee-fc94-4a61-8cde-343483e43de6 2020-09-08T17:01:43.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 8c36d4ee-fc94-4a61-8cde-343483e43de6' 2020-09-08T17:01:43.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:01:43.802 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD3uFdfG0QlMBAAr7kQWu/7GmURHyaNtlmUOg== 2020-09-08T17:01:43.802 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD3uFdfG0QlMBAAr7kQWu/7GmURHyaNtlmUOg=="}' 2020-09-08T17:01:43.803 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8c36d4ee-fc94-4a61-8cde-343483e43de6 -i td/test-erasure-eio/2/new.json 2020-09-08T17:01:44.751 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:01:44.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:01:44.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD3uFdfG0QlMBAAr7kQWu/7GmURHyaNtlmUOg== --osd-uuid 8c36d4ee-fc94-4a61-8cde-343483e43de6 2020-09-08T17:01:44.770 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:44.774+0000 7faee1a38ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:44.771 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:44.776+0000 7faee1a38ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:44.773 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:44.778+0000 7faee1a38ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:44.774 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:44.779+0000 7faee1a38ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:01:44.774 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:44.779+0000 7faee1a38ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:01:44.774 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:44.779+0000 7faee1a38ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:01:44.819 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:44.825+0000 7faee1a38ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:01:49.803 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:01:49.803 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:01:49.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:01:49.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:01:49.805 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:01:50.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:01:50.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:50.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:01:50.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:01:50.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:01:50.322 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:01:50.343 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:50.349+0000 7fb6d6055ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:50.353 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:50.359+0000 7fb6d6055ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:50.355 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:50.360+0000 7fb6d6055ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:50.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:01:50.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:01:50.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:01:50.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:01:50.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:01:50.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:50.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:01:50.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:50.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:01:50.696 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:01:50.871 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:50.876+0000 7fb6d6055ec0 -1 Falling back to public interface 2020-09-08T17:01:51.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:01:51.754 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:51.759+0000 7fb6d6055ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:01:52.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:01:52.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:52.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:01:52.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:52.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:01:52.092 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:01:52.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:01:53.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:01:53.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:01:53.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:01:53.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:01:53.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:01:53.529 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:01:54.047 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/112688,v1:127.0.0.1:6819/112688] [v2:127.0.0.1:6820/112688,v1:127.0.0.1:6821/112688] exists,up 8c36d4ee-fc94-4a61-8cde-343483e43de6 2020-09-08T17:01:54.047 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 c43ddca8-f8df-402e-965f-98f005eff225 2020-09-08T17:01:54.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:01:54.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:01:54.048 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:01:54.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:01:54.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:01:54.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:01:54.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:01:54.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:01:54.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:01:54.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:01:54.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:01:54.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:01:54.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:01:54.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:01:54.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:01:54.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:01:54.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:01:54.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:01:54.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:01:54.052 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:01:54.052 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:01:54.052 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:01:54.052 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:01:54.053 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:01:54.053 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:01:54.053 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:54.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:01:54.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:01:54.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:01:54.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:01:54.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:01:54.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:01:54.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:01:54.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:01:54.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:01:54.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:01:54.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:01:54.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:01:54.056 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:01:54.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c43ddca8-f8df-402e-965f-98f005eff225 2020-09-08T17:01:54.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 c43ddca8-f8df-402e-965f-98f005eff225' 2020-09-08T17:01:54.056 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:01:54.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQACuVdfRqIKBBAABbfesVnyT6Q0p+OtuMZfrQ== 2020-09-08T17:01:54.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQACuVdfRqIKBBAABbfesVnyT6Q0p+OtuMZfrQ=="}' 2020-09-08T17:01:54.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c43ddca8-f8df-402e-965f-98f005eff225 -i td/test-erasure-eio/3/new.json 2020-09-08T17:01:54.664 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:01:54.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:01:54.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQACuVdfRqIKBBAABbfesVnyT6Q0p+OtuMZfrQ== --osd-uuid c43ddca8-f8df-402e-965f-98f005eff225 2020-09-08T17:01:54.699 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:54.705+0000 7f289964dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:54.700 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:54.706+0000 7f289964dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:54.701 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:54.707+0000 7f289964dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:01:54.702 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:54.708+0000 7f289964dec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:01:54.702 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:54.708+0000 7f289964dec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:01:54.703 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:54.708+0000 7f289964dec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:01:54.848 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:01:54.854+0000 7f289964dec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:01:59.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:01:59.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:01:59.643 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:01:59.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:01:59.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:02:00.144 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:02:00.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:02:00.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:00.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:02:00.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:02:00.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:02:00.166 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:00.172+0000 7f91a12ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:00.168 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:00.174+0000 7f91a12ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:00.170 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:00.175+0000 7f91a12ddec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:00.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:02:00.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:02:00.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:02:00.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:02:00.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:02:00.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:00.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:02:00.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:00.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:02:00.521 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:02:00.696 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:00.702+0000 7f91a12ddec0 -1 Falling back to public interface 2020-09-08T17:02:00.892 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:02:01.516 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:01.520+0000 7f91a12ddec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:02:01.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:02:01.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:01.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:02:01.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:01.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:02:01.895 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:02:02.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:02:03.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:02:03.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:03.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:02:03.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:03.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:02:03.311 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:02:03.712 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/113708,v1:127.0.0.1:6827/113708] [v2:127.0.0.1:6828/113708,v1:127.0.0.1:6829/113708] exists,up c43ddca8-f8df-402e-965f-98f005eff225 2020-09-08T17:02:03.712 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 717ea631-5ba6-4b04-a4f3-2807b2c311cb 2020-09-08T17:02:03.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:02:03.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:02:03.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:02:03.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:02:03.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 4 2020-09-08T17:02:03.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:02:03.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:02:03.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T17:02:03.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:02:03.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/4 2020-09-08T17:02:03.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:02:03.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:02:03.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:02:03.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:02:03.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2020-09-08T17:02:03.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2020-09-08T17:02:03.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:02:03.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:02:03.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:02:03.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:02:03.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:02:03.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:02:03.718 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:02:03.719 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:03.719 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:03.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:03.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:03.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:02:03.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:02:03.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:02:03.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:02:03.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:02:03.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:02:03.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:02:03.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:03.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:02:03.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:02:03.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/4 2020-09-08T17:02:03.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:02:03.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=717ea631-5ba6-4b04-a4f3-2807b2c311cb 2020-09-08T17:02:03.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 717ea631-5ba6-4b04-a4f3-2807b2c311cb' 2020-09-08T17:02:03.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:02:03.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQALuVdfBk6hKxAAT+CA7SKtY58RusAKUZW20w== 2020-09-08T17:02:03.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQALuVdfBk6hKxAAT+CA7SKtY58RusAKUZW20w=="}' 2020-09-08T17:02:03.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 717ea631-5ba6-4b04-a4f3-2807b2c311cb -i td/test-erasure-eio/4/new.json 2020-09-08T17:02:04.326 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T17:02:04.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/4/new.json 2020-09-08T17:02:04.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQALuVdfBk6hKxAAT+CA7SKtY58RusAKUZW20w== --osd-uuid 717ea631-5ba6-4b04-a4f3-2807b2c311cb 2020-09-08T17:02:04.362 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:04.366+0000 7fccedb69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:04.363 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:04.368+0000 7fccedb69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:04.364 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:04.369+0000 7fccedb69ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:04.365 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:04.370+0000 7fccedb69ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:02:04.365 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:04.370+0000 7fccedb69ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:02:04.365 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:04.370+0000 7fccedb69ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:02:04.427 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:04.431+0000 7fccedb69ec0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2020-09-08T17:02:08.987 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2020-09-08T17:02:08.987 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:02:08.992 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T17:02:08.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T17:02:08.992 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:02:09.486 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T17:02:09.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T17:02:09.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:09.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:02:09.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:02:09.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:02:09.508 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:09.513+0000 7f483250fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:09.510 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:09.515+0000 7f483250fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:09.512 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:09.516+0000 7f483250fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:09.889 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T17:02:09.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:02:09.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T17:02:09.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:02:09.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:02:09.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:09.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:02:09.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:09.891 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:02:09.892 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:02:10.029 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:10.033+0000 7f483250fec0 -1 Falling back to public interface 2020-09-08T17:02:10.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:02:10.845 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:10.850+0000 7f483250fec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T17:02:11.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:02:11.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:11.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:02:11.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:11.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:02:11.268 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:02:11.676 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/114728,v1:127.0.0.1:6835/114728] [v2:127.0.0.1:6836/114728,v1:127.0.0.1:6837/114728] exists,up 717ea631-5ba6-4b04-a4f3-2807b2c311cb 2020-09-08T17:02:11.677 INFO:tasks.workunit.client.0.smithi120.stdout:add osd5 6cf6e7e8-46be-4836-ae2e-b790cca78947 2020-09-08T17:02:11.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:02:11.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:02:11.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:02:11.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:02:11.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 5 2020-09-08T17:02:11.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:02:11.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:02:11.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-09-08T17:02:11.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:02:11.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/5 2020-09-08T17:02:11.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:02:11.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:02:11.681 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:02:11.681 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:02:11.681 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2020-09-08T17:02:11.681 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2020-09-08T17:02:11.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:02:11.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:02:11.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:02:11.682 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:02:11.683 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:02:11.683 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:02:11.683 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:02:11.683 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:11.684 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:11.684 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:11.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:11.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:02:11.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:02:11.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:02:11.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:02:11.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:02:11.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:02:11.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:02:11.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:11.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:02:11.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:02:11.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/5 2020-09-08T17:02:11.686 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:02:11.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6cf6e7e8-46be-4836-ae2e-b790cca78947 2020-09-08T17:02:11.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 6cf6e7e8-46be-4836-ae2e-b790cca78947' 2020-09-08T17:02:11.687 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:02:11.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQATuVdfrrBlKRAAOkeXCl3VETSrP2Zim8XpZw== 2020-09-08T17:02:11.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQATuVdfrrBlKRAAOkeXCl3VETSrP2Zim8XpZw=="}' 2020-09-08T17:02:11.689 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6cf6e7e8-46be-4836-ae2e-b790cca78947 -i td/test-erasure-eio/5/new.json 2020-09-08T17:02:12.432 INFO:tasks.workunit.client.0.smithi120.stdout:5 2020-09-08T17:02:12.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/5/new.json 2020-09-08T17:02:12.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQATuVdfrrBlKRAAOkeXCl3VETSrP2Zim8XpZw== --osd-uuid 6cf6e7e8-46be-4836-ae2e-b790cca78947 2020-09-08T17:02:12.467 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:12.472+0000 7fbb11909ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:12.469 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:12.474+0000 7fbb11909ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:12.470 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:12.475+0000 7fbb11909ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:12.471 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:12.475+0000 7fbb11909ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:02:12.471 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:12.476+0000 7fbb11909ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:02:12.471 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:12.476+0000 7fbb11909ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:02:12.624 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:12.630+0000 7fbb11909ec0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2020-09-08T17:02:18.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2020-09-08T17:02:18.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:02:18.257 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd5 key to auth repository 2020-09-08T17:02:18.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-09-08T17:02:18.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:02:18.755 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.5 2020-09-08T17:02:18.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-09-08T17:02:18.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:18.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:02:18.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:02:18.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:02:18.777 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:18.782+0000 7fd3fb1fdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:18.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:18.784+0000 7fd3fb1fdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:18.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:18.785+0000 7fd3fb1fdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:19.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-09-08T17:02:19.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:02:19.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-09-08T17:02:19.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:02:19.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:02:19.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:19.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:02:19.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:19.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:02:19.159 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:02:19.296 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:19.300+0000 7fd3fb1fdec0 -1 Falling back to public interface 2020-09-08T17:02:19.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:02:20.205 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:20.210+0000 7fd3fb1fdec0 -1 osd.5 0 log_to_monitors {default=true} 2020-09-08T17:02:20.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:02:20.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:20.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:02:20.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:20.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:02:20.537 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:02:21.040 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:02:22.042 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:02:22.042 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:22.042 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:02:22.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:22.043 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:02:22.043 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:02:22.477 INFO:tasks.workunit.client.0.smithi120.stdout:osd.5 up in weight 1 up_from 39 up_thru 39 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/115722,v1:127.0.0.1:6843/115722] [v2:127.0.0.1:6844/115722,v1:127.0.0.1:6845/115722] exists,up 6cf6e7e8-46be-4836-ae2e-b790cca78947 2020-09-08T17:02:22.477 INFO:tasks.workunit.client.0.smithi120.stdout:add osd6 add76ba9-f33a-49c1-ae56-a3bb8b9d0860 2020-09-08T17:02:22.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:02:22.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:02:22.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:02:22.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:02:22.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 6 2020-09-08T17:02:22.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:02:22.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:02:22.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2020-09-08T17:02:22.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:02:22.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/6 2020-09-08T17:02:22.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:02:22.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:02:22.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:02:22.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:02:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2020-09-08T17:02:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2020-09-08T17:02:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:02:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:02:22.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:02:22.482 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:02:22.482 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:02:22.482 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:02:22.482 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:02:22.483 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:22.483 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:22.483 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:22.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:22.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:02:22.484 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:02:22.484 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:02:22.484 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:02:22.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:02:22.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:02:22.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:02:22.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:22.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:02:22.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:02:22.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/6 2020-09-08T17:02:22.486 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:02:22.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=add76ba9-f33a-49c1-ae56-a3bb8b9d0860 2020-09-08T17:02:22.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 add76ba9-f33a-49c1-ae56-a3bb8b9d0860' 2020-09-08T17:02:22.487 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:02:22.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAeuVdf4FKvHRAAC7goTMesCw2gIMLEq/OiMA== 2020-09-08T17:02:22.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAeuVdf4FKvHRAAC7goTMesCw2gIMLEq/OiMA=="}' 2020-09-08T17:02:22.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new add76ba9-f33a-49c1-ae56-a3bb8b9d0860 -i td/test-erasure-eio/6/new.json 2020-09-08T17:02:22.989 INFO:tasks.workunit.client.0.smithi120.stdout:6 2020-09-08T17:02:22.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/6/new.json 2020-09-08T17:02:23.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAeuVdf4FKvHRAAC7goTMesCw2gIMLEq/OiMA== --osd-uuid add76ba9-f33a-49c1-ae56-a3bb8b9d0860 2020-09-08T17:02:23.024 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:23.029+0000 7f9226d1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:23.025 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:23.030+0000 7f9226d1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:23.026 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:23.031+0000 7f9226d1fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:23.027 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:23.032+0000 7f9226d1fec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:02:23.027 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:23.032+0000 7f9226d1fec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:02:23.027 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:23.032+0000 7f9226d1fec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:02:23.141 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:23.146+0000 7f9226d1fec0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2020-09-08T17:02:28.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2020-09-08T17:02:28.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:02:28.472 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd6 key to auth repository 2020-09-08T17:02:28.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2020-09-08T17:02:28.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:02:28.970 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.6 2020-09-08T17:02:28.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2020-09-08T17:02:28.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:28.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:02:28.971 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:02:28.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:02:28.996 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:28.998+0000 7f0730277ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:28.996 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:28.999+0000 7f0730277ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:28.997 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:29.001+0000 7f0730277ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:29.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2020-09-08T17:02:29.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:02:29.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-09-08T17:02:29.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:02:29.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:02:29.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:29.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:02:29.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:29.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:02:29.343 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:02:29.512 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:29.517+0000 7f0730277ec0 -1 Falling back to public interface 2020-09-08T17:02:29.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:02:30.455 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:30.461+0000 7f0730277ec0 -1 osd.6 0 log_to_monitors {default=true} 2020-09-08T17:02:30.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:02:30.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:30.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:02:30.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:30.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:02:30.751 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:02:31.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:02:32.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:02:32.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:02:32.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:02:32.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:02:32.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:02:32.132 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:02:32.590 INFO:tasks.workunit.client.0.smithi120.stdout:osd.6 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/116742,v1:127.0.0.1:6851/116742] [v2:127.0.0.1:6852/116742,v1:127.0.0.1:6853/116742] exists,up add76ba9-f33a-49c1-ae56-a3bb8b9d0860 2020-09-08T17:02:32.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:02:32.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:02:32.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:02:32.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:02:32.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:02:32.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:02:32.591 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:02:32.592 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:32.592 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:32.592 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:02:32.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:02:32.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:02:32.681 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:02:32.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:02:32.691 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:01:33.620+0000 7f03206faec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:02:32.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:02:32.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:487: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2020-09-08T17:02:32.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2020-09-08T17:02:32.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:02:32.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:02:32.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=3 2020-09-08T17:02:32.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:02:32.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=2 2020-09-08T17:02:32.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:02:32.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2020-09-08T17:02:33.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:02:33.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:02:34.047 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:02:34.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:02:35.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:02:35.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:02:35.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:02:35.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:02:35.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:02:35.058 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:02:35.059 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:02:35.059 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:02:35.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:02:35.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:02:35.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:02:35.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:02:35.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:02:35.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:02:35.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:02:35.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:02:35.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:02:35.529 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:02:35.529 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:02:35.530 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:02:35.530 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:02:35.530 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:02:35.530 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:02:35.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:02:35.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:35.531 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:02:35.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803790 2020-09-08T17:02:35.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803790 2020-09-08T17:02:35.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790' 2020-09-08T17:02:35.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:35.668 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:02:35.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574861 2020-09-08T17:02:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574861 2020-09-08T17:02:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861' 2020-09-08T17:02:35.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:35.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:02:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378635 2020-09-08T17:02:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378635 2020-09-08T17:02:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635' 2020-09-08T17:02:35.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:35.943 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:02:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149705 2020-09-08T17:02:36.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149705 2020-09-08T17:02:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705' 2020-09-08T17:02:36.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:36.063 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:02:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920776 2020-09-08T17:02:36.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920776 2020-09-08T17:02:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705 4-141733920776' 2020-09-08T17:02:36.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:36.206 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T17:02:36.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724549 2020-09-08T17:02:36.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724549 2020-09-08T17:02:36.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705 4-141733920776 5-167503724549' 2020-09-08T17:02:36.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:36.382 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T17:02:36.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528323 2020-09-08T17:02:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528323 2020-09-08T17:02:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803790 1-55834574861 2-81604378635 3-111669149705 4-141733920776 5-167503724549 6-193273528323' 2020-09-08T17:02:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:36.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803790 2020-09-08T17:02:36.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:36.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:02:36.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803790 2020-09-08T17:02:36.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:36.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803790 2020-09-08T17:02:36.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803790' 2020-09-08T17:02:36.506 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:02:36.506 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803790 2020-09-08T17:02:36.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803790 2020-09-08T17:02:36.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:02:37.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:02:37.868 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:02:38.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803791 -lt 25769803790 2020-09-08T17:02:38.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:38.280 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574861 2020-09-08T17:02:38.280 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:38.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:02:38.281 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574861 2020-09-08T17:02:38.281 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:38.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574861 2020-09-08T17:02:38.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574861' 2020-09-08T17:02:38.282 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:02:38.283 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574861 2020-09-08T17:02:38.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574861 -lt 55834574861 2020-09-08T17:02:38.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:38.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378635 2020-09-08T17:02:38.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:38.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:02:38.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378635 2020-09-08T17:02:38.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:38.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378635 2020-09-08T17:02:38.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378635' 2020-09-08T17:02:38.649 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:02:38.649 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378635 2020-09-08T17:02:39.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378635 -lt 81604378635 2020-09-08T17:02:39.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:39.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149705 2020-09-08T17:02:39.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:39.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:02:39.016 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149705 2020-09-08T17:02:39.016 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:39.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149705 2020-09-08T17:02:39.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149705' 2020-09-08T17:02:39.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:02:39.017 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149705 2020-09-08T17:02:39.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149705 -lt 111669149705 2020-09-08T17:02:39.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:39.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920776 2020-09-08T17:02:39.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:39.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:02:39.404 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920776 2020-09-08T17:02:39.404 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:39.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920776 2020-09-08T17:02:39.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920776' 2020-09-08T17:02:39.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:02:39.406 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920776 2020-09-08T17:02:39.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920777 -lt 141733920776 2020-09-08T17:02:39.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:39.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-167503724549 2020-09-08T17:02:39.797 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:39.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T17:02:39.797 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-167503724549 2020-09-08T17:02:39.798 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:39.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724549 2020-09-08T17:02:39.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 167503724549' 2020-09-08T17:02:39.800 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T17:02:39.800 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 167503724549 2020-09-08T17:02:40.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724550 -lt 167503724549 2020-09-08T17:02:40.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:40.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-193273528323 2020-09-08T17:02:40.216 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:40.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T17:02:40.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-193273528323 2020-09-08T17:02:40.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:40.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528323 2020-09-08T17:02:40.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 193273528323' 2020-09-08T17:02:40.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T17:02:40.218 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 193273528323 2020-09-08T17:02:40.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528324 -lt 193273528323 2020-09-08T17:02:40.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:02:40.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:02:40.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:02:41.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:02:41.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:02:41.083 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:02:41.083 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:02:41.084 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:02:41.084 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:02:41.084 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:02:41.084 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:02:41.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:02:41.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:02:41.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:02:41.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:02:41.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:02:41.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:02:41.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:02:41.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:490: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2020-09-08T17:02:41.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:02:41.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:02:41.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=test1 2020-09-08T17:02:41.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:41.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:02:41.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:41.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:02:41.947 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:41.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:02:41.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:41.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:02:41.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2020-09-08T17:02:42.101 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2020-09-08T17:02:42.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:02:42.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:02:42.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=test2 2020-09-08T17:02:42.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:02:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:02:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:02:42.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:02:42.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2020-09-08T17:02:42.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2020-09-08T17:02:42.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:02:42.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:02:42.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=test3 2020-09-08T17:02:42.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:02:42.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:02:42.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:02:42.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:02:42.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:02:42.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2020-09-08T17:02:42.698 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:495: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2020-09-08T17:02:42.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:02:42.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:02:42.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:02:42.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:02:42.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:02:42.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:02:42.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=myobject 2020-09-08T17:02:42.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:02:42.700 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:02:42.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:02:42.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2020-09-08T17:02:42.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:02:42.701 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:02:42.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure myobject 2020-09-08T17:02:42.701 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:02:42.702 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T17:02:42.702 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T17:02:42.702 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:02:43.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:02:43.071 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:02:43.072 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:02:43.072 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:02:43.072 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:02:43.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 1 0 6 2020-09-08T17:02:43.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:02:43.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2020-09-08T17:02:43.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:02:43.073 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/3/type 2020-09-08T17:02:43.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:02:43.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2020-09-08T17:02:43.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:02:43.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2020-09-08T17:02:43.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:02:43.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:02:43.074 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:02:43.075 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2020-09-08T17:02:43.075 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:02:43.075 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:02:43.075 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:02:43.075 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:43.076 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:43.076 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:02:43.076 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:02:43.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:02:43.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:02:43.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:02:43.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2020-09-08T17:02:43.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:02:43.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:02:43.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:02:43.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:43.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:43.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:02:43.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:02:43.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2020-09-08T17:02:43.261 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2020-09-08T17:02:43.785 INFO:tasks.workunit.client.0.smithi120.stderr:osd.0 is already out. 2020-09-08T17:02:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:499: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2020-09-08T17:02:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:02:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:02:43.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:02:43.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:02:43.795 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:02:43.796 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:02:43.796 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:02:43.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:02:43.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:02:43.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:02:43.885 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:02:43.886 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:02:43.886 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:02:43.886 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:02:43.886 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:02:44.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:02:44.339 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:02:44.339 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:02:44.339 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:02:44.339 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:02:44.340 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:02:44.340 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:02:44.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:02:44.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:44.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:02:44.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803794 2020-09-08T17:02:44.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803794 2020-09-08T17:02:44.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794' 2020-09-08T17:02:44.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:44.566 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:02:44.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574865 2020-09-08T17:02:44.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574865 2020-09-08T17:02:44.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574865' 2020-09-08T17:02:44.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:44.739 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:02:45.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378639 2020-09-08T17:02:45.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378639 2020-09-08T17:02:45.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574865 2-81604378639' 2020-09-08T17:02:45.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:45.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:02:45.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149709 2020-09-08T17:02:45.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149709 2020-09-08T17:02:45.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574865 2-81604378639 3-111669149709' 2020-09-08T17:02:45.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:45.138 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:02:45.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920780 2020-09-08T17:02:45.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920780 2020-09-08T17:02:45.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574865 2-81604378639 3-111669149709 4-141733920780' 2020-09-08T17:02:45.259 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:45.259 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T17:02:45.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724553 2020-09-08T17:02:45.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724553 2020-09-08T17:02:45.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574865 2-81604378639 3-111669149709 4-141733920780 5-167503724553' 2020-09-08T17:02:45.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:02:45.381 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T17:02:45.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528327 2020-09-08T17:02:45.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528327 2020-09-08T17:02:45.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-55834574865 2-81604378639 3-111669149709 4-141733920780 5-167503724553 6-193273528327' 2020-09-08T17:02:45.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:45.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803794 2020-09-08T17:02:45.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:45.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:02:45.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803794 2020-09-08T17:02:45.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:45.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803794 2020-09-08T17:02:45.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803794' 2020-09-08T17:02:45.505 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803794 2020-09-08T17:02:45.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:02:46.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803794 -lt 25769803794 2020-09-08T17:02:46.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:46.008 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574865 2020-09-08T17:02:46.009 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:46.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:02:46.009 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574865 2020-09-08T17:02:46.010 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:46.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574865 2020-09-08T17:02:46.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574865' 2020-09-08T17:02:46.011 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:02:46.011 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574865 2020-09-08T17:02:46.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574865 -lt 55834574865 2020-09-08T17:02:46.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:46.414 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378639 2020-09-08T17:02:46.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:46.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:02:46.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378639 2020-09-08T17:02:46.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:46.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378639 2020-09-08T17:02:46.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2020-09-08T17:02:46.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:02:46.417 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378639 2020-09-08T17:02:46.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378639 -lt 81604378639 2020-09-08T17:02:46.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:46.812 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149709 2020-09-08T17:02:46.812 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:46.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:02:46.813 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149709 2020-09-08T17:02:46.813 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:46.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149709 2020-09-08T17:02:46.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149709' 2020-09-08T17:02:46.815 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:02:46.815 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149709 2020-09-08T17:02:47.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149709 -lt 111669149709 2020-09-08T17:02:47.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:47.177 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920780 2020-09-08T17:02:47.178 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:47.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:02:47.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920780 2020-09-08T17:02:47.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:47.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920780 2020-09-08T17:02:47.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920780' 2020-09-08T17:02:47.180 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:02:47.180 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920780 2020-09-08T17:02:47.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920780 -lt 141733920780 2020-09-08T17:02:47.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:47.545 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-167503724553 2020-09-08T17:02:47.546 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:47.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T17:02:47.546 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-167503724553 2020-09-08T17:02:47.547 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:47.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724553 2020-09-08T17:02:47.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 167503724553' 2020-09-08T17:02:47.548 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T17:02:47.548 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 167503724553 2020-09-08T17:02:47.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724554 -lt 167503724553 2020-09-08T17:02:47.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:02:47.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-193273528327 2020-09-08T17:02:47.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:02:47.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T17:02:47.960 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-193273528327 2020-09-08T17:02:47.960 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:02:47.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528327 2020-09-08T17:02:47.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 193273528327' 2020-09-08T17:02:47.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T17:02:47.962 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 193273528327 2020-09-08T17:02:48.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528328 -lt 193273528327 2020-09-08T17:02:48.365 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:02:48.365 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:02:48.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:02:48.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:02:48.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:02:48.844 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:02:48.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:02:48.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:02:48.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:02:48.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:02:48.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:02:49.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-09-08T17:02:49.195 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:02:49.195 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:02:49.195 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:02:49.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 5 2020-09-08T17:02:49.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 '!=' -1 2020-09-08T17:02:49.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T17:02:49.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=3 2020-09-08T17:02:49.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:02:49.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T17:02:49.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:02:49.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:02:49.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:02:49.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:02:49.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:02:49.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:02:49.793 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:02:49.794 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:02:50.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:02:50.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:02:50.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:02:50.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:02:50.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:02:50.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:02:50.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:02:50.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:501: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2020-09-08T17:02:50.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:02:50.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:02:50.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=test1 2020-09-08T17:02:50.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:02:50.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:02:50.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2020-09-08T17:02:50.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:02:50.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:02:50.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2020-09-08T17:02:50.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:02:50.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:02:50.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=test2 2020-09-08T17:02:50.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:02:50.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:02:50.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2020-09-08T17:02:50.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:02:50.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:02:50.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2020-09-08T17:02:50.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:02:50.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:02:50.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=test3 2020-09-08T17:02:50.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:02:50.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:02:50.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2020-09-08T17:02:50.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:02:50.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:02:50.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:505: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:02:50.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:02:50.792 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:02:51.318 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:02:51.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:02:52.197 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:02:52.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:02:52.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:02:52.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:02:52.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:02:52.206 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:02:52.206 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:02:52.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:02:52.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:02:52.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:02:52.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:02:52.316 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:02:52.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:02:52.320 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:02:52.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:02:52.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:02:52.324 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:02:52.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:02:52.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:02:52.324 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:02:52.325 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:02:52.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:02:52.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:02:52.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:02:52.331 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:02:52.331 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:52.331 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:52.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:02:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:02:52.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:02:52.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:02:52.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:02:52.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:02:52.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:02:52.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:02:52.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:02:52.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:02:52.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:02:52.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:02:52.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:02:52.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:02:52.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:02:52.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:02:52.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:02:52.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:02:52.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:02:52.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:02:52.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:02:52.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:02:52.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:02:52.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:02:52.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:02:52.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:02:52.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:02:52.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:02:52.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:02:52.347 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:02:52.347 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:52.347 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:52.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:02:52.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:02:52.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:02:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:02:52.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:02:52.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:52.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:02:52.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:02:52.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:02:52.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:02:52.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:02:52.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:02:52.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:02:52.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:02:52.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:02:52.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:02:52.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:02:52.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:02:52.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:02:52.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:02:52.696 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:02:52.696 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:02:52.696 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:52.696 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:52.697 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:52.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:02:52.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:02:52.889 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:02:52.890 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:02:52.890 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:02:52.890 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:02:52.890 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:02:52.890 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:02:52.891 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:02:52.891 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:02:52.891 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:02:52.891 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:52.891 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:52.892 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:02:52.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:02:52.892 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:02:52.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:02:52.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:02:52.978 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:02:52.978 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:02:52.978 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:02:52.978 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:02:52.978 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:02:52.979 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:02:52.980 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:02:52.980 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:52.980 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:52.980 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:02:52.981 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:02:52.981 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:02:53.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:02:53.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:02:53.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:02:53.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:02:53.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:02:53.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:02:53.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:02:53.948 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:02:53.949 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:02:53.949 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:02:53.949 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:02:53.949 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:53.949 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:53.950 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:53.950 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:02:53.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:02:53.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:02:53.970 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:02:54.265 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:02:54.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:02:55.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:02:55.276 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:02:55.276 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:02:55.276 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:02:55.276 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:55.276 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:55.277 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:02:55.277 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:02:55.277 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:02:55.356 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:02:55.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:02:55.365 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:02:52.728+0000 7f44c671f6c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:02:55.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_ec_recovery_unfound td/test-erasure-eio 2020-09-08T17:02:55.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2020-09-08T17:02:55.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:590: TEST_ec_recovery_unfound: local objname=myobject 2020-09-08T17:02:55.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_recovery_unfound: local lastobj=100 2020-09-08T17:02:55.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_recovery_unfound: local testobj=obj75 2020-09-08T17:02:55.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:02:55.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:596: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2020-09-08T17:02:55.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T17:02:55.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_recovery_unfound: setup_osds 5 2020-09-08T17:02:55.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=5 2020-09-08T17:02:55.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:02:55.368 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 5 - 1 2020-09-08T17:02:55.372 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 4 2020-09-08T17:02:55.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:02:55.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:02:55.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:02:55.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:02:55.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:02:55.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:02:55.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:02:55.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T17:02:55.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:02:55.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:02:55.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:02:55.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:02:55.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:02:55.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:02:55.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:02:55.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:02:55.375 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:02:55.376 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:02:55.376 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:02:55.376 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:02:55.376 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:02:55.376 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:02:55.376 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:55.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:02:55.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:02:55.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:02:55.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:02:55.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:02:55.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:02:55.378 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:02:55.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:02:55.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:02:55.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:02:55.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:02:55.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:02:55.380 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:02:55.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=689b9782-a717-43c9-8e86-d0f1572c0edd 2020-09-08T17:02:55.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 689b9782-a717-43c9-8e86-d0f1572c0edd' 2020-09-08T17:02:55.381 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:02:55.381 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 689b9782-a717-43c9-8e86-d0f1572c0edd 2020-09-08T17:02:55.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA/uVdfm9mLFxAALDdcyfmJnzfjYtYrkKLYGg== 2020-09-08T17:02:55.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA/uVdfm9mLFxAALDdcyfmJnzfjYtYrkKLYGg=="}' 2020-09-08T17:02:55.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 689b9782-a717-43c9-8e86-d0f1572c0edd -i td/test-erasure-eio/0/new.json 2020-09-08T17:02:55.731 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:02:55.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:02:55.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA/uVdfm9mLFxAALDdcyfmJnzfjYtYrkKLYGg== --osd-uuid 689b9782-a717-43c9-8e86-d0f1572c0edd 2020-09-08T17:02:55.770 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:55.775+0000 7f82853abec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:55.772 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:55.777+0000 7f82853abec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:55.773 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:55.778+0000 7f82853abec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:02:55.774 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:55.779+0000 7f82853abec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:02:55.774 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:55.779+0000 7f82853abec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:02:55.774 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:55.779+0000 7f82853abec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:02:55.835 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:02:55.840+0000 7f82853abec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:03:00.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:03:00.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:03:00.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:03:00.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:03:00.499 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:03:01.002 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:03:01.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:03:01.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:03:01.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:01.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:03:01.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:03:01.028 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:01.033+0000 7fa12d22bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:01.029 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:01.034+0000 7fa12d22bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:01.031 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:01.036+0000 7fa12d22bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:01.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:03:01.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:03:01.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:03:01.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:03:01.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:03:01.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:01.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:03:01.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:01.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:03:01.418 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:03:01.547 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:01.551+0000 7fa12d22bec0 -1 Falling back to public interface 2020-09-08T17:03:01.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:03:02.390 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:02.395+0000 7fa12d22bec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:03:02.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:03:02.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:02.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:03:02.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:02.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:03:02.788 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:03:03.185 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/119639,v1:127.0.0.1:6803/119639] [v2:127.0.0.1:6804/119639,v1:127.0.0.1:6805/119639] exists,up 689b9782-a717-43c9-8e86-d0f1572c0edd 2020-09-08T17:03:03.185 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 0903f65e-e977-4511-8633-b9c1246dfdd9 2020-09-08T17:03:03.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:03:03.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:03:03.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:03:03.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:03:03.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:03:03.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:03:03.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:03:03.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:03:03.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:03:03.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:03:03.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T17:03:03.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:03:03.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:03:03.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:03:03.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:03:03.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:03:03.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:03:03.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:03:03.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:03:03.192 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:03:03.192 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:03:03.192 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:03:03.192 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:03:03.192 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:03:03.193 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:03:03.193 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:03.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:03.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:03:03.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:03:03.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:03:03.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:03:03.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:03:03.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:03:03.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:03:03.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:03.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:03:03.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:03:03.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:03:03.203 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:03:03.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0903f65e-e977-4511-8633-b9c1246dfdd9 2020-09-08T17:03:03.204 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 0903f65e-e977-4511-8633-b9c1246dfdd9' 2020-09-08T17:03:03.204 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:03:03.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBHuVdfNxZvDBAAhGdB1ujpnECoVxhhFcdkXw== 2020-09-08T17:03:03.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBHuVdfNxZvDBAAhGdB1ujpnECoVxhhFcdkXw=="}' 2020-09-08T17:03:03.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0903f65e-e977-4511-8633-b9c1246dfdd9 -i td/test-erasure-eio/1/new.json 2020-09-08T17:03:04.116 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:03:04.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:03:04.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBHuVdfNxZvDBAAhGdB1ujpnECoVxhhFcdkXw== --osd-uuid 0903f65e-e977-4511-8633-b9c1246dfdd9 2020-09-08T17:03:04.155 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:04.160+0000 7fb281cdaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:04.166 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:04.171+0000 7fb281cdaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:04.167 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:04.172+0000 7fb281cdaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:04.168 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:04.172+0000 7fb281cdaec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:03:04.168 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:04.172+0000 7fb281cdaec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:03:04.168 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:04.172+0000 7fb281cdaec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:03:04.224 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:04.230+0000 7fb281cdaec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:03:08.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:03:08.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:03:08.959 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:03:08.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:03:08.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:03:09.463 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:03:09.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:03:09.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:03:09.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:09.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:03:09.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:03:09.489 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:09.494+0000 7f2d28ea7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:09.491 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:09.496+0000 7f2d28ea7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:09.492 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:09.497+0000 7f2d28ea7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:09.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:03:09.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:03:09.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:03:09.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:03:09.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:03:09.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:09.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:03:09.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:09.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:03:09.843 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:03:10.015 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:10.019+0000 7f2d28ea7ec0 -1 Falling back to public interface 2020-09-08T17:03:10.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:03:10.854 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:10.859+0000 7f2d28ea7ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:03:11.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:03:11.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:11.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:03:11.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:11.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:03:11.214 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:03:11.614 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/120633,v1:127.0.0.1:6811/120633] [v2:127.0.0.1:6812/120633,v1:127.0.0.1:6813/120633] exists,up 0903f65e-e977-4511-8633-b9c1246dfdd9 2020-09-08T17:03:11.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:03:11.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:03:11.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:03:11.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:03:11.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:03:11.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:03:11.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:03:11.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:03:11.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:03:11.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:03:11.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T17:03:11.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:03:11.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:03:11.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:03:11.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:03:11.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:03:11.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:03:11.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:03:11.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:03:11.619 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:03:11.619 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:03:11.619 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:03:11.619 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:03:11.619 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:03:11.620 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:03:11.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:11.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:11.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:03:11.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:03:11.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:03:11.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:03:11.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:03:11.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:03:11.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:03:11.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:11.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:03:11.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:03:11.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:03:11.622 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:03:11.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b292bba4-7842-4cdc-b27f-92f998b179f9 2020-09-08T17:03:11.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 b292bba4-7842-4cdc-b27f-92f998b179f9' 2020-09-08T17:03:11.623 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:03:11.624 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 b292bba4-7842-4cdc-b27f-92f998b179f9 2020-09-08T17:03:11.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBPuVdf7JsaJhAARyExODuGxKXlAB3FX7FMUQ== 2020-09-08T17:03:11.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBPuVdf7JsaJhAARyExODuGxKXlAB3FX7FMUQ=="}' 2020-09-08T17:03:11.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b292bba4-7842-4cdc-b27f-92f998b179f9 -i td/test-erasure-eio/2/new.json 2020-09-08T17:03:12.483 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:03:12.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:03:12.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBPuVdf7JsaJhAARyExODuGxKXlAB3FX7FMUQ== --osd-uuid b292bba4-7842-4cdc-b27f-92f998b179f9 2020-09-08T17:03:12.523 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:12.527+0000 7fd33b949ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:12.524 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:12.529+0000 7fd33b949ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:12.525 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:12.530+0000 7fd33b949ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:12.526 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:12.531+0000 7fd33b949ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:03:12.526 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:12.531+0000 7fd33b949ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:03:12.526 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:12.531+0000 7fd33b949ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:03:12.588 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:12.593+0000 7fd33b949ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:03:17.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:03:17.466 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:03:17.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:03:17.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:03:17.467 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:03:17.971 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:03:17.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:03:17.972 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:03:17.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:17.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:03:17.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:03:17.998 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:18.004+0000 7fb8e08c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:17.999 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:18.006+0000 7fb8e08c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:18.001 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:18.007+0000 7fb8e08c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:18.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:03:18.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:03:18.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:03:18.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:03:18.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:03:18.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:18.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:03:18.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:18.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:03:18.349 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:03:18.520 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:18.526+0000 7fb8e08c6ec0 -1 Falling back to public interface 2020-09-08T17:03:18.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:03:19.345 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:19.351+0000 7fb8e08c6ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:03:19.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:03:19.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:19.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:03:19.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:19.743 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:03:19.743 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:03:20.232 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/121627,v1:127.0.0.1:6819/121627] [v2:127.0.0.1:6820/121627,v1:127.0.0.1:6821/121627] exists,up b292bba4-7842-4cdc-b27f-92f998b179f9 2020-09-08T17:03:20.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:03:20.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:03:20.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:03:20.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:03:20.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:03:20.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:03:20.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:03:20.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:03:20.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:03:20.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:03:20.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T17:03:20.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:03:20.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:03:20.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:03:20.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:03:20.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:03:20.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:03:20.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:03:20.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:03:20.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:03:20.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:03:20.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:03:20.238 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:03:20.239 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:03:20.239 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:03:20.239 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:20.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:20.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:03:20.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:03:20.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:03:20.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:03:20.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:03:20.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:03:20.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:03:20.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:20.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:03:20.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:03:20.241 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:03:20.244 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:03:20.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7640d315-e446-4475-8633-cbee4adc37b6 2020-09-08T17:03:20.245 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 7640d315-e446-4475-8633-cbee4adc37b6' 2020-09-08T17:03:20.246 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:03:20.246 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 7640d315-e446-4475-8633-cbee4adc37b6 2020-09-08T17:03:20.250 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBYuVdfzetIDxAAYwIrmTSrUKsQYWMnJVUFmg== 2020-09-08T17:03:20.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBYuVdfzetIDxAAYwIrmTSrUKsQYWMnJVUFmg=="}' 2020-09-08T17:03:20.251 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7640d315-e446-4475-8633-cbee4adc37b6 -i td/test-erasure-eio/3/new.json 2020-09-08T17:03:21.223 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:03:21.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:03:21.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBYuVdfzetIDxAAYwIrmTSrUKsQYWMnJVUFmg== --osd-uuid 7640d315-e446-4475-8633-cbee4adc37b6 2020-09-08T17:03:21.261 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:21.267+0000 7f5ad0765ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:21.262 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:21.269+0000 7f5ad0765ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:21.263 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:21.269+0000 7f5ad0765ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:21.264 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:21.270+0000 7f5ad0765ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:03:21.264 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:21.270+0000 7f5ad0765ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:03:21.264 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:21.270+0000 7f5ad0765ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:03:21.336 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:21.342+0000 7f5ad0765ec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:03:26.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:03:26.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:03:26.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:03:26.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:03:26.753 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:03:27.256 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:03:27.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:03:27.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:27.257 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:03:27.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:03:27.258 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:03:27.283 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:27.289+0000 7facc920cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:27.284 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:27.290+0000 7facc920cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:27.285 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:27.292+0000 7facc920cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:27.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:03:27.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:03:27.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:03:27.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:03:27.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:03:27.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:27.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:03:27.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:27.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:03:27.638 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:03:27.802 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:27.806+0000 7facc920cec0 -1 Falling back to public interface 2020-09-08T17:03:28.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:03:28.657 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:28.662+0000 7facc920cec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:03:29.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:03:29.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:29.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:03:29.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:29.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:03:29.018 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:03:29.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:03:30.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:03:30.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:30.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:03:30.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:30.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:03:30.509 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:03:31.017 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/122621,v1:127.0.0.1:6827/122621] [v2:127.0.0.1:6828/122621,v1:127.0.0.1:6829/122621] exists,up 7640d315-e446-4475-8633-cbee4adc37b6 2020-09-08T17:03:31.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:03:31.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:03:31.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:03:31.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:03:31.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 4 2020-09-08T17:03:31.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:03:31.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:03:31.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T17:03:31.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:03:31.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/4 2020-09-08T17:03:31.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2020-09-08T17:03:31.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:03:31.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:03:31.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:03:31.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2020-09-08T17:03:31.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2020-09-08T17:03:31.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:03:31.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:03:31.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:03:31.022 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:03:31.022 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:03:31.022 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:03:31.022 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:03:31.022 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:03:31.023 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:03:31.023 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:31.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:03:31.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:03:31.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:03:31.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:03:31.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:03:31.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:03:31.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:03:31.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:03:31.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:31.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:03:31.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:03:31.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/4 2020-09-08T17:03:31.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:03:31.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2894a7f6-670a-4508-bc96-764d6c72ae61 2020-09-08T17:03:31.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 2894a7f6-670a-4508-bc96-764d6c72ae61' 2020-09-08T17:03:31.027 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:03:31.027 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 2894a7f6-670a-4508-bc96-764d6c72ae61 2020-09-08T17:03:31.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBjuVdfjdR6AhAA4GdCgRpRUNCU8mtZCL5+Cw== 2020-09-08T17:03:31.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBjuVdfjdR6AhAA4GdCgRpRUNCU8mtZCL5+Cw=="}' 2020-09-08T17:03:31.036 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2894a7f6-670a-4508-bc96-764d6c72ae61 -i td/test-erasure-eio/4/new.json 2020-09-08T17:03:31.766 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T17:03:31.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/4/new.json 2020-09-08T17:03:31.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBjuVdfjdR6AhAA4GdCgRpRUNCU8mtZCL5+Cw== --osd-uuid 2894a7f6-670a-4508-bc96-764d6c72ae61 2020-09-08T17:03:31.804 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:31.809+0000 7fda6f040ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:31.806 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:31.811+0000 7fda6f040ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:31.807 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:31.812+0000 7fda6f040ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:31.808 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:31.813+0000 7fda6f040ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:03:31.808 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:31.813+0000 7fda6f040ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:03:31.808 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:31.813+0000 7fda6f040ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:03:31.892 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:31.897+0000 7fda6f040ec0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2020-09-08T17:03:36.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2020-09-08T17:03:36.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:03:36.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T17:03:36.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:03:36.408 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T17:03:36.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T17:03:36.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:03:36.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:03:36.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:03:36.905 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T17:03:36.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:03:36.930 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:36.936+0000 7f482143dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:36.932 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:36.937+0000 7f482143dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:36.934 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:36.939+0000 7f482143dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:03:37.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T17:03:37.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:03:37.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T17:03:37.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:03:37.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:03:37.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:37.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:03:37.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:37.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:03:37.303 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:03:37.454 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:37.459+0000 7f482143dec0 -1 Falling back to public interface 2020-09-08T17:03:37.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:03:38.366 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:03:38.370+0000 7f482143dec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T17:03:38.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:03:38.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:38.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:03:38.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:03:38.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:38.684 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:03:39.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:03:40.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:03:40.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:03:40.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:03:40.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:03:40.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:03:40.089 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:03:40.547 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 33 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/123649,v1:127.0.0.1:6835/123649] [v2:127.0.0.1:6836/123649,v1:127.0.0.1:6837/123649] exists,up 2894a7f6-670a-4508-bc96-764d6c72ae61 2020-09-08T17:03:40.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:03:40.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:03:40.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:03:40.548 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:03:40.548 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:03:40.548 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:03:40.549 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:03:40.550 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:03:40.550 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:03:40.550 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:03:40.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:03:40.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:03:40.638 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:03:40.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:03:40.648 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:03:01.558+0000 7fa12d22bec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:03:40.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:599: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:03:40.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:601: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2020-09-08T17:03:40.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2020-09-08T17:03:40.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:03:40.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:03:40.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=3 2020-09-08T17:03:40.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:03:40.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=2 2020-09-08T17:03:40.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:03:40.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2020-09-08T17:03:41.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:03:41.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:03:42.112 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:03:42.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:03:43.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:03:43.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:03:43.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:03:43.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:03:43.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:03:43.123 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:03:43.124 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:03:43.124 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:03:43.124 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:03:43.124 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:03:43.125 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:03:43.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:03:43.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:03:43.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:03:43.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:03:43.227 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:03:43.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:03:43.598 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:03:43.598 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:03:43.598 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:03:43.599 INFO:tasks.workunit.client.0.smithi120.stderr:4' 2020-09-08T17:03:43.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:03:43.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:03:43.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:03:43.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803787 2020-09-08T17:03:43.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803787 2020-09-08T17:03:43.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787' 2020-09-08T17:03:43.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:03:43.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:03:43.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574858 2020-09-08T17:03:43.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574858 2020-09-08T17:03:43.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787 1-55834574858' 2020-09-08T17:03:43.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:03:43.883 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:03:44.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378631 2020-09-08T17:03:44.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378631 2020-09-08T17:03:44.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-81604378631' 2020-09-08T17:03:44.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:03:44.003 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:03:44.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149701 2020-09-08T17:03:44.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149701 2020-09-08T17:03:44.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-81604378631 3-111669149701' 2020-09-08T17:03:44.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:03:44.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:03:44.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920771 2020-09-08T17:03:44.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920771 2020-09-08T17:03:44.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-81604378631 3-111669149701 4-141733920771' 2020-09-08T17:03:44.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:03:44.263 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803787 2020-09-08T17:03:44.264 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:03:44.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:03:44.264 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803787 2020-09-08T17:03:44.264 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:03:44.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803787 2020-09-08T17:03:44.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2020-09-08T17:03:44.265 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:03:44.266 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803787 2020-09-08T17:03:44.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803785 -lt 25769803787 2020-09-08T17:03:44.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:03:45.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:03:45.626 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:03:46.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803787 2020-09-08T17:03:46.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:03:46.005 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574858 2020-09-08T17:03:46.005 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:03:46.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:03:46.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574858 2020-09-08T17:03:46.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:03:46.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574858 2020-09-08T17:03:46.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574858' 2020-09-08T17:03:46.008 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:03:46.008 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574858 2020-09-08T17:03:46.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574858 -lt 55834574858 2020-09-08T17:03:46.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:03:46.371 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378631 2020-09-08T17:03:46.371 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:03:46.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:03:46.372 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378631 2020-09-08T17:03:46.372 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:03:46.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378631 2020-09-08T17:03:46.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378631' 2020-09-08T17:03:46.374 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:03:46.374 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378631 2020-09-08T17:03:46.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378631 -lt 81604378631 2020-09-08T17:03:46.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:03:46.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149701 2020-09-08T17:03:46.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:03:46.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:03:46.737 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149701 2020-09-08T17:03:46.737 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:03:46.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149701 2020-09-08T17:03:46.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149701' 2020-09-08T17:03:46.738 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:03:46.739 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149701 2020-09-08T17:03:47.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149702 -lt 111669149701 2020-09-08T17:03:47.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:03:47.140 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920771 2020-09-08T17:03:47.140 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:03:47.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:03:47.141 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920771 2020-09-08T17:03:47.141 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:03:47.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920771 2020-09-08T17:03:47.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920771' 2020-09-08T17:03:47.142 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920771 2020-09-08T17:03:47.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:03:47.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920772 -lt 141733920771 2020-09-08T17:03:47.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:03:47.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:03:47.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:03:47.989 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:03:47.990 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:03:47.990 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:03:47.990 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:03:47.991 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:03:47.991 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:03:47.991 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:03:47.991 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:03:48.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:03:48.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:03:48.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:03:48.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:03:48.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:03:48.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:03:48.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:03:48.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_recovery_unfound: ceph pg dump pgs 2020-09-08T17:03:49.195 INFO:tasks.workunit.client.0.smithi120.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-09-08T17:03:49.195 INFO:tasks.workunit.client.0.smithi120.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:43.722763+0000 0'0 39:11 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2020-09-08T17:03:41.772582+0000 0'0 2020-09-08T17:03:41.772582+0000 0 2020-09-08T17:03:49.196 INFO:tasks.workunit.client.0.smithi120.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.158799+0000 0'0 39:15 [4,1,2] 4 [4,1,2] 4 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:49.196 INFO:tasks.workunit.client.0.smithi120.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.661145+0000 0'0 38:63 [0,4,1] 0 [0,4,1] 0 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:49.196 INFO:tasks.workunit.client.0.smithi120.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:21.062678+0000 0'0 39:36 [1,0,2] 1 [1,0,2] 1 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:49.196 INFO:tasks.workunit.client.0.smithi120.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.284049+0000 0'0 39:15 [4,3,0] 4 [4,3,0] 4 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:49.196 INFO:tasks.workunit.client.0.smithi120.stdout: 2020-09-08T17:03:49.197 INFO:tasks.workunit.client.0.smithi120.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-09-08T17:03:49.197 INFO:tasks.workunit.client.0.smithi120.stderr:dumped pgs 2020-09-08T17:03:49.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:606: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2020-09-08T17:03:49.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:03:49.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:03:49.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=myobject 2020-09-08T17:03:49.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:03:49.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:03:49.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:03:49.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:03:49.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:03:49.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:03:49.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:03:49.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:03:49.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:49.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:608: TEST_ec_recovery_unfound: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:03:49.393 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:608: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2020-09-08T17:03:49.393 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:03:49.393 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T17:03:49.394 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T17:03:49.394 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:03:49.769 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:03:49.769 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:03:49.770 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:03:49.770 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:03:49.770 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:03:49.770 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 4 0 2 2020-09-08T17:03:49.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:608: TEST_ec_recovery_unfound: local -a initial_osds 2020-09-08T17:03:49.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local last_osd=2 2020-09-08T17:03:49.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2020-09-08T17:03:49.771 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:03:49.771 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:03:49.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:03:49.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:03:49.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:03:49.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:03:49.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: ceph osd down 2 2020-09-08T17:03:50.719 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 is already down. 2020-09-08T17:03:50.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:612: TEST_ec_recovery_unfound: ceph osd out 2 2020-09-08T17:03:51.418 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 is already out. 2020-09-08T17:03:51.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:614: TEST_ec_recovery_unfound: ceph pg dump pgs 2020-09-08T17:03:51.822 INFO:tasks.workunit.client.0.smithi120.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-09-08T17:03:51.822 INFO:tasks.workunit.client.0.smithi120.stdout:2.0 1 0 0 0 0 4096 0 0 1 1 active+clean 2020-09-08T17:03:43.722763+0000 40'1 40:13 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2020-09-08T17:03:41.772582+0000 0'0 2020-09-08T17:03:41.772582+0000 0 2020-09-08T17:03:51.822 INFO:tasks.workunit.client.0.smithi120.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.158799+0000 0'0 39:15 [4,1,2] 4 [4,1,2] 4 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:51.822 INFO:tasks.workunit.client.0.smithi120.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.661145+0000 0'0 39:64 [0,4,1] 0 [0,4,1] 0 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:51.823 INFO:tasks.workunit.client.0.smithi120.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:21.062678+0000 0'0 39:36 [1,0,2] 1 [1,0,2] 1 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:51.823 INFO:tasks.workunit.client.0.smithi120.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.284049+0000 0'0 39:15 [4,3,0] 4 [4,3,0] 4 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:03:51.823 INFO:tasks.workunit.client.0.smithi120.stdout: 2020-09-08T17:03:51.823 INFO:tasks.workunit.client.0.smithi120.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-09-08T17:03:51.824 INFO:tasks.workunit.client.0.smithi120.stderr:dumped pgs 2020-09-08T17:03:51.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2020-09-08T17:03:51.832 INFO:tasks.workunit.client.0.smithi120.stderr:4+0 records in 2020-09-08T17:03:51.832 INFO:tasks.workunit.client.0.smithi120.stderr:4+0 records out 2020-09-08T17:03:51.833 INFO:tasks.workunit.client.0.smithi120.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 8.598e-05 s, 47.6 MB/s 2020-09-08T17:03:51.833 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: seq 1 100 2020-09-08T17:03:51.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:51.834 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:52.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:52.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:52.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:52.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:52.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:52.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:53.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:53.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:53.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:53.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:53.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:53.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:54.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:54.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:54.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:54.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:54.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:54.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:54.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:54.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:55.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:55.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:55.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:55.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:55.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:55.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:55.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:55.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:56.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:56.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:56.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:56.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:56.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:56.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:56.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:56.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:57.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:57.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:57.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:57.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:57.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:57.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:57.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:57.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:58.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:58.175 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:58.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:58.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:58.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:58.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:58.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:58.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:59.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:59.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:59.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:59.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:59.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:59.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:59.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:59.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2020-09-08T17:03:59.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:03:59.879 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:00.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:00.105 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:00.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:00.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:00.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:00.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:00.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:00.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:00.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:00.895 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:01.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:01.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:01.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:01.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:01.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:01.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:01.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:01.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:01.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:01.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:02.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:02.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:02.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:02.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:02.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:02.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:02.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:02.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:02.993 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:02.993 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:03.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:03.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:03.379 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:03.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:03.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:03.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:03.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:03.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:04.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:04.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:04.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:04.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:04.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:04.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:04.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:04.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:04.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:04.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:05.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:05.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:05.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:05.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:05.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:05.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:05.679 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:05.680 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:05.889 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:05.890 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:06.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:06.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:06.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:06.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:06.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:06.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:06.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:06.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:06.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:06.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:07.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:07.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:07.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:07.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:07.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:07.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:07.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:07.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:07.979 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:07.980 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:08.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:08.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:08.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:08.416 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:08.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:08.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:08.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:08.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:09.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:09.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:09.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:09.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:09.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:09.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:09.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:09.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:09.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:09.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:10.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:10.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:10.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:10.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:10.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:10.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:10.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:10.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:10.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:10.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:11.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:11.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:11.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:11.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:11.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:11.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:11.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:11.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:11.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:11.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:12.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:12.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:12.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:12.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:12.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:12.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:12.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:12.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:12.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:12.969 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:13.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:13.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:13.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:13.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:13.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:13.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:13.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:13.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:14.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:14.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2020-09-08T17:04:14.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2020-09-08T17:04:14.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:04:14.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:04:14.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:04:14.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:04:14.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:04:14.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:04:14.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj75 2020-09-08T17:04:14.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:04:14.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:04:14.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:04:14.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2020-09-08T17:04:14.233 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:04:14.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:04:14.234 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj75 2020-09-08T17:04:14.234 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:04:14.234 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj75 2020-09-08T17:04:14.234 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj75 2020-09-08T17:04:14.234 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:04:14.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:04:14.614 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:04:14.614 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:04:14.615 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:04:14.615 INFO:tasks.workunit.client.0.smithi120.stderr:2147483647' 2020-09-08T17:04:14.615 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 4 0 2147483647 2020-09-08T17:04:14.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:04:14.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2020-09-08T17:04:14.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:04:14.616 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/3/type 2020-09-08T17:04:14.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:04:14.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2020-09-08T17:04:14.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:04:14.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2020-09-08T17:04:14.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:04:14.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:04:14.619 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:04:14.619 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2020-09-08T17:04:14.619 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:04:14.619 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:04:14.620 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:04:14.620 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:14.620 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:14.620 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:04:14.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:04:14.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:04:14.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:04:14.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:04:14.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2020-09-08T17:04:14.715 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:04:14.715 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:04:14.716 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:04:14.716 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:14.717 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:14.717 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:04:14.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:04:14.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2020-09-08T17:04:14.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:623: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2020-09-08T17:04:14.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:04:14.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:04:14.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:04:14.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:04:14.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:04:14.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:04:14.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj75 2020-09-08T17:04:14.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:04:14.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:04:14.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:04:14.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2020-09-08T17:04:14.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:04:14.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:04:14.812 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj75 2020-09-08T17:04:14.813 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:04:14.813 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj75 2020-09-08T17:04:14.813 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj75 2020-09-08T17:04:14.813 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:04:15.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:04:15.185 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:04:15.185 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:04:15.185 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:04:15.185 INFO:tasks.workunit.client.0.smithi120.stderr:2147483647' 2020-09-08T17:04:15.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 4 0 2147483647 2020-09-08T17:04:15.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:04:15.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2020-09-08T17:04:15.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:04:15.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/1/type 2020-09-08T17:04:15.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:04:15.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-09-08T17:04:15.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:04:15.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2020-09-08T17:04:15.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:04:15.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:04:15.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:04:15.189 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2020-09-08T17:04:15.190 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T17:04:15.190 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T17:04:15.190 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:04:15.190 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:15.190 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:15.191 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T17:04:15.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:04:15.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:04:15.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:04:15.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:04:15.283 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2020-09-08T17:04:15.283 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T17:04:15.283 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T17:04:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:04:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:15.284 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T17:04:15.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:04:15.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2020-09-08T17:04:15.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:625: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2020-09-08T17:04:15.381 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:04:15.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:04:15.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-09-08T17:04:15.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:04:15.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:04:15.382 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:04:15.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:04:15.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:04:15.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:04:15.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:04:15.383 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:04:15.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:04:15.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:04:15.384 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:04:15.384 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:04:15.384 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:04:15.385 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:04:15.385 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:04:15.385 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:15.385 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:15.386 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:15.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:15.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:04:15.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:04:15.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:04:15.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:04:15.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:04:15.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:04:15.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:04:15.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:04:15.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:04:15.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-09-08T17:04:15.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:04:15.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/2/whoami 2020-09-08T17:04:15.388 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:04:15.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-09-08T17:04:15.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:04:15.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:04:15.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:04:15.399 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:15.405+0000 7fd4298f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:15.401 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:15.406+0000 7fd4298f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:15.402 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:15.407+0000 7fd4298f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:15.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-09-08T17:04:15.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:04:15.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:04:15.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:04:15.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:04:15.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:04:15.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:04:15.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:04:15.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:04:15.750 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:04:15.920 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:15.925+0000 7fd4298f9ec0 -1 Falling back to public interface 2020-09-08T17:04:16.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:04:17.083 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:17.088+0000 7fd4298f9ec0 -1 osd.2 40 log_to_monitors {default=true} 2020-09-08T17:04:17.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:04:17.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:04:17.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:04:17.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:04:17.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:04:17.142 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:04:17.547 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up out weight 0 up_from 47 up_thru 19 down_at 41 last_clean_interval [19,40) [v2:127.0.0.1:6818/126865,v1:127.0.0.1:6819/126865] [v2:127.0.0.1:6820/126865,v1:127.0.0.1:6821/126865] exists,up b292bba4-7842-4cdc-b27f-92f998b179f9 2020-09-08T17:04:17.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:04:17.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:04:17.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:04:17.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: ceph osd in 2 2020-09-08T17:04:18.408 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 is already in. 2020-09-08T17:04:18.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: sleep 15 2020-09-08T17:04:33.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: seq 1 100 2020-09-08T17:04:33.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2020-09-08T17:04:33.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: get_state 2.0 2020-09-08T17:04:33.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:59: get_state: local pgid=2.0 2020-09-08T17:04:33.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local sname=state 2020-09-08T17:04:33.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: ceph --format json pg dump pgs 2020-09-08T17:04:33.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2020-09-08T17:04:33.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2020-09-08T17:04:33.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2020-09-08T17:04:33.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: grep recovery_unfound 2020-09-08T17:04:33.787 INFO:tasks.workunit.client.0.smithi120.stdout:active+recovery_unfound+degraded 2020-09-08T17:04:33.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:633: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2020-09-08T17:04:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: break 2020-09-08T17:04:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:640: TEST_ec_recovery_unfound: ceph pg dump pgs 2020-09-08T17:04:34.126 INFO:tasks.workunit.client.0.smithi120.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-09-08T17:04:34.126 INFO:tasks.workunit.client.0.smithi120.stdout:2.0 101 1 3 0 1 413696 0 0 101 101 active+recovery_unfound+degraded 2020-09-08T17:04:28.698438+0000 46'101 52:348 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2020-09-08T17:03:41.772582+0000 0'0 2020-09-08T17:03:41.772582+0000 0 2020-09-08T17:04:34.126 INFO:tasks.workunit.client.0.smithi120.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:04:19.106444+0000 0'0 51:51 [4,1,2] 4 [4,1,2] 4 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:04:34.127 INFO:tasks.workunit.client.0.smithi120.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.661145+0000 0'0 50:74 [0,4,1] 0 [0,4,1] 0 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:04:34.127 INFO:tasks.workunit.client.0.smithi120.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:04:20.301235+0000 0'0 52:72 [1,0,2] 1 [1,0,2] 1 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:04:34.127 INFO:tasks.workunit.client.0.smithi120.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2020-09-08T17:03:40.284049+0000 0'0 51:27 [4,3,0] 4 [4,3,0] 4 0'0 2020-09-08T17:02:54.141296+0000 0'0 2020-09-08T17:02:54.141296+0000 0 2020-09-08T17:04:34.127 INFO:tasks.workunit.client.0.smithi120.stdout: 2020-09-08T17:04:34.128 INFO:tasks.workunit.client.0.smithi120.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-09-08T17:04:34.128 INFO:tasks.workunit.client.0.smithi120.stderr:dumped pgs 2020-09-08T17:04:34.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:641: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2020-09-08T17:04:34.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:641: TEST_ec_recovery_unfound: grep -q obj75 2020-09-08T17:04:34.261 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:644: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2020-09-08T17:04:39.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: test 124 = 124 2020-09-08T17:04:39.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:647: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2020-09-08T17:04:39.536 INFO:tasks.workunit.client.0.smithi120.stderr:pg has no unfound objects 2020-09-08T17:04:39.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:649: TEST_ec_recovery_unfound: wait_for_clean 2020-09-08T17:04:39.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:04:39.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:04:39.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:04:39.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:04:39.547 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:04:39.548 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:04:39.548 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:04:39.548 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:04:39.549 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:04:39.549 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:04:39.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:04:39.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:04:39.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:04:39.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:04:39.639 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:04:40.071 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:04:40.071 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:04:40.072 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:04:40.072 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:04:40.072 INFO:tasks.workunit.client.0.smithi120.stderr:4' 2020-09-08T17:04:40.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:04:40.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:04:40.074 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:04:40.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803801 2020-09-08T17:04:40.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803801 2020-09-08T17:04:40.192 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801' 2020-09-08T17:04:40.193 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:04:40.193 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:04:40.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574871 2020-09-08T17:04:40.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574871 2020-09-08T17:04:40.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801 1-55834574871' 2020-09-08T17:04:40.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:04:40.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:04:40.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=201863462919 2020-09-08T17:04:40.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 201863462919 2020-09-08T17:04:40.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801 1-55834574871 2-201863462919' 2020-09-08T17:04:40.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:04:40.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:04:40.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149715 2020-09-08T17:04:40.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149715 2020-09-08T17:04:40.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801 1-55834574871 2-201863462919 3-111669149715' 2020-09-08T17:04:40.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:04:40.552 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:04:40.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=141733920785 2020-09-08T17:04:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 141733920785 2020-09-08T17:04:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801 1-55834574871 2-201863462919 3-111669149715 4-141733920785' 2020-09-08T17:04:40.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:04:40.673 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803801 2020-09-08T17:04:40.673 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:04:40.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:04:40.674 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803801 2020-09-08T17:04:40.675 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:04:40.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803801 2020-09-08T17:04:40.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2020-09-08T17:04:40.675 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:04:40.675 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803801 2020-09-08T17:04:41.037 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803799 -lt 25769803801 2020-09-08T17:04:41.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:04:42.038 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:04:42.039 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:04:42.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803801 -lt 25769803801 2020-09-08T17:04:42.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:04:42.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574871 2020-09-08T17:04:42.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:04:42.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:04:42.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574871 2020-09-08T17:04:42.416 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:04:42.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574871 2020-09-08T17:04:42.417 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2020-09-08T17:04:42.417 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:04:42.417 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574871 2020-09-08T17:04:42.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574871 -lt 55834574871 2020-09-08T17:04:42.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:04:42.785 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-201863462919 2020-09-08T17:04:42.786 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:04:42.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:04:42.786 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-201863462919 2020-09-08T17:04:42.787 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:04:42.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=201863462919 2020-09-08T17:04:42.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 201863462919' 2020-09-08T17:04:42.788 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:04:42.788 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 201863462919 2020-09-08T17:04:43.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 201863462919 -lt 201863462919 2020-09-08T17:04:43.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:04:43.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149715 2020-09-08T17:04:43.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:04:43.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:04:43.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149715 2020-09-08T17:04:43.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:04:43.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149715 2020-09-08T17:04:43.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149715' 2020-09-08T17:04:43.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:04:43.152 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149715 2020-09-08T17:04:43.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149715 -lt 111669149715 2020-09-08T17:04:43.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:04:43.517 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-141733920785 2020-09-08T17:04:43.517 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:04:43.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:04:43.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-141733920785 2020-09-08T17:04:43.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:04:43.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=141733920785 2020-09-08T17:04:43.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 141733920785' 2020-09-08T17:04:43.519 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:04:43.519 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 141733920785 2020-09-08T17:04:43.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 141733920785 -lt 141733920785 2020-09-08T17:04:43.882 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:04:43.882 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:04:43.882 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:04:44.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:04:44.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:04:44.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:04:44.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:04:44.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:04:44.388 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:04:44.389 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:04:44.389 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:04:44.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:04:44.739 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:04:44.740 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:04:44.740 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:04:45.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:04:45.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:04:45.254 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:04:45.255 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: seq 1 100 2020-09-08T17:04:45.255 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2020-09-08T17:04:45.256 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2020-09-08T17:04:45.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2020-09-08T17:04:45.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2020-09-08T17:04:45.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2020-09-08T17:04:45.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2020-09-08T17:04:45.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2020-09-08T17:04:45.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2020-09-08T17:04:45.451 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2020-09-08T17:04:45.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.498 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2020-09-08T17:04:45.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2020-09-08T17:04:45.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2020-09-08T17:04:45.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2020-09-08T17:04:45.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2020-09-08T17:04:45.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2020-09-08T17:04:45.643 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.667 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2020-09-08T17:04:45.668 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2020-09-08T17:04:45.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2020-09-08T17:04:45.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2020-09-08T17:04:45.742 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2020-09-08T17:04:45.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2020-09-08T17:04:45.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2020-09-08T17:04:45.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2020-09-08T17:04:45.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2020-09-08T17:04:45.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.887 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2020-09-08T17:04:45.888 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.911 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2020-09-08T17:04:45.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2020-09-08T17:04:45.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2020-09-08T17:04:45.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2020-09-08T17:04:45.985 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:45.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:45.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2020-09-08T17:04:45.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2020-09-08T17:04:46.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2020-09-08T17:04:46.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.067 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2020-09-08T17:04:46.068 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2020-09-08T17:04:46.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2020-09-08T17:04:46.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2020-09-08T17:04:46.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2020-09-08T17:04:46.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2020-09-08T17:04:46.196 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2020-09-08T17:04:46.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2020-09-08T17:04:46.240 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2020-09-08T17:04:46.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2020-09-08T17:04:46.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2020-09-08T17:04:46.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2020-09-08T17:04:46.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2020-09-08T17:04:46.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2020-09-08T17:04:46.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2020-09-08T17:04:46.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2020-09-08T17:04:46.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2020-09-08T17:04:46.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2020-09-08T17:04:46.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2020-09-08T17:04:46.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.534 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2020-09-08T17:04:46.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2020-09-08T17:04:46.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2020-09-08T17:04:46.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2020-09-08T17:04:46.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2020-09-08T17:04:46.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2020-09-08T17:04:46.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.681 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.682 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2020-09-08T17:04:46.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.707 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.707 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2020-09-08T17:04:46.707 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2020-09-08T17:04:46.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2020-09-08T17:04:46.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2020-09-08T17:04:46.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.805 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.805 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2020-09-08T17:04:46.805 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2020-09-08T17:04:46.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2020-09-08T17:04:46.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2020-09-08T17:04:46.878 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2020-09-08T17:04:46.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2020-09-08T17:04:46.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2020-09-08T17:04:46.952 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:46.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:46.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2020-09-08T17:04:46.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2020-09-08T17:04:46.999 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.000 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2020-09-08T17:04:47.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2020-09-08T17:04:47.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2020-09-08T17:04:47.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2020-09-08T17:04:47.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.098 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2020-09-08T17:04:47.099 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2020-09-08T17:04:47.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:655: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.144 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2020-09-08T17:04:47.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2020-09-08T17:04:47.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2020-09-08T17:04:47.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.194 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2020-09-08T17:04:47.195 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2020-09-08T17:04:47.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2020-09-08T17:04:47.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.267 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.268 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2020-09-08T17:04:47.269 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2020-09-08T17:04:47.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2020-09-08T17:04:47.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2020-09-08T17:04:47.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2020-09-08T17:04:47.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.389 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2020-09-08T17:04:47.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.414 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2020-09-08T17:04:47.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2020-09-08T17:04:47.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2020-09-08T17:04:47.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2020-09-08T17:04:47.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2020-09-08T17:04:47.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.537 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2020-09-08T17:04:47.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2020-09-08T17:04:47.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2020-09-08T17:04:47.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2020-09-08T17:04:47.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2020-09-08T17:04:47.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.683 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2020-09-08T17:04:47.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2020-09-08T17:04:47.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2020-09-08T17:04:47.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2020-09-08T17:04:47.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2020-09-08T17:04:47.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:657: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2020-09-08T17:04:47.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:658: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2020-09-08T17:04:47.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:04:47.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:04:47.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:04:48.299 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:04:48.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:04:49.005 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:04:49.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:04:49.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:04:49.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:04:49.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:04:49.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:04:49.015 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:04:49.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:04:49.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:04:49.016 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:04:49.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:04:49.124 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:04:49.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:04:49.124 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:04:49.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:04:49.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:04:49.126 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:04:49.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:04:49.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:04:49.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:04:49.128 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:04:49.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:04:49.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:04:49.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:04:49.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:04:49.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:49.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:49.143 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:04:49.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:04:49.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:04:49.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:04:49.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:04:49.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:04:49.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:04:49.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:04:49.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:04:49.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:04:49.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:04:49.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:04:49.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:04:49.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:04:49.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:04:49.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:04:49.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:04:49.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:04:49.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:04:49.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:04:49.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:04:49.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:04:49.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:04:49.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:04:49.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:04:49.152 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:04:49.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:04:49.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:04:49.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:04:49.154 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:04:49.154 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:49.154 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:49.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:04:49.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:04:49.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:04:49.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:04:49.156 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:04:49.156 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:49.156 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:49.156 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:04:49.158 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:04:49.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:04:49.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:04:49.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:04:49.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:04:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:04:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:04:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:04:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:04:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:04:49.160 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:04:49.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:04:49.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:04:49.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:04:49.426 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:04:49.426 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:49.427 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:49.427 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:49.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:04:49.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:04:49.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:04:49.654 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:04:49.654 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:04:49.654 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:04:49.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:04:49.655 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:04:49.655 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:04:49.655 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:04:49.655 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:04:49.656 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:49.656 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:49.656 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:04:49.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:04:49.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:04:49.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:04:49.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:04:49.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:04:49.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:04:49.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:04:49.743 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:04:49.743 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:04:49.743 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:04:49.743 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:04:49.743 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:49.744 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:49.744 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:04:49.744 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:04:49.744 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:04:49.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:04:49.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:04:49.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:04:49.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:04:49.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:04:49.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:04:49.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:04:50.710 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:04:50.711 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:04:50.711 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:04:50.711 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:04:50.711 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:50.712 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:50.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:50.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:04:50.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:04:50.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:04:50.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:04:51.037 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:04:51.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:04:52.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:04:52.048 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:04:52.048 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:04:52.048 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:04:52.048 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:52.049 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:52.049 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:04:52.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:04:52.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:04:52.128 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:04:52.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:04:52.138 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:04:49.460+0000 7f1ce00086c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:04:52.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_ec_single_recovery_error td/test-erasure-eio 2020-09-08T17:04:52.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:389: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2020-09-08T17:04:52.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local objname=myobject 2020-09-08T17:04:52.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:392: TEST_ec_single_recovery_error: setup_osds 7 2020-09-08T17:04:52.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=7 2020-09-08T17:04:52.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:04:52.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 7 - 1 2020-09-08T17:04:52.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 6 2020-09-08T17:04:52.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:04:52.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:04:52.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:04:52.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:04:52.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:04:52.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:04:52.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:04:52.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:04:52.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:04:52.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:04:52.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:04:52.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:04:52.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:04:52.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:04:52.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:04:52.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:04:52.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:04:52.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:04:52.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:04:52.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:04:52.148 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:52.148 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:52.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:52.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:52.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:04:52.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:04:52.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:04:52.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:04:52.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:04:52.149 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:04:52.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:04:52.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:04:52.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:04:52.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:04:52.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:04:52.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:04:52.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f6bea66c-afc3-4924-aa79-785cb628de1f 2020-09-08T17:04:52.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 f6bea66c-afc3-4924-aa79-785cb628de1f' 2020-09-08T17:04:52.152 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:04:52.152 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 f6bea66c-afc3-4924-aa79-785cb628de1f 2020-09-08T17:04:52.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC0uVdfyIrBCRAAgQYr3OuU9Pd0uRJKEc7aJQ== 2020-09-08T17:04:52.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC0uVdfyIrBCRAAgQYr3OuU9Pd0uRJKEc7aJQ=="}' 2020-09-08T17:04:52.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f6bea66c-afc3-4924-aa79-785cb628de1f -i td/test-erasure-eio/0/new.json 2020-09-08T17:04:52.486 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:04:52.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:04:52.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC0uVdfyIrBCRAAgQYr3OuU9Pd0uRJKEc7aJQ== --osd-uuid f6bea66c-afc3-4924-aa79-785cb628de1f 2020-09-08T17:04:52.517 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:52.524+0000 7ff350aadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:52.519 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:52.525+0000 7ff350aadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:52.520 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:52.527+0000 7ff350aadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:52.521 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:52.527+0000 7ff350aadec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:04:52.521 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:52.527+0000 7ff350aadec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:04:52.521 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:52.527+0000 7ff350aadec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:04:52.552 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:52.558+0000 7ff350aadec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:04:56.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:04:56.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:04:56.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:04:56.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:04:56.505 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:04:56.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:04:56.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:04:56.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:04:56.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:04:56.997 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:04:56.997 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:04:57.014 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:57.020+0000 7f9646e39ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:57.016 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:57.022+0000 7f9646e39ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:57.017 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:57.022+0000 7f9646e39ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:57.364 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:04:57.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:04:57.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:04:57.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:04:57.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:04:57.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:04:57.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:04:57.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:04:57.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:04:57.367 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:04:57.532 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:57.537+0000 7f9646e39ec0 -1 Falling back to public interface 2020-09-08T17:04:57.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:04:58.310 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:58.315+0000 7f9646e39ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:04:58.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:04:58.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:04:58.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:04:58.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:04:58.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:04:58.733 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:04:59.212 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/130984,v1:127.0.0.1:6803/130984] [v2:127.0.0.1:6804/130984,v1:127.0.0.1:6805/130984] exists,up f6bea66c-afc3-4924-aa79-785cb628de1f 2020-09-08T17:04:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:04:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:04:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:04:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:04:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:04:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:04:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:04:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:04:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:04:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:04:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:04:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:04:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:04:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:04:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:04:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:04:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:04:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:04:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:04:59.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:04:59.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:04:59.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:04:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:04:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:04:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:04:59.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:04:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:04:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:04:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:04:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:04:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:04:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:04:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:04:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:04:59.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:04:59.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:04:59.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:04:59.220 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:04:59.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ea876442-f5ea-4950-a3b2-b9feb220b85c 2020-09-08T17:04:59.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ea876442-f5ea-4950-a3b2-b9feb220b85c' 2020-09-08T17:04:59.222 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:04:59.222 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 ea876442-f5ea-4950-a3b2-b9feb220b85c 2020-09-08T17:04:59.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC7uVdfLlbVDRAAW7ZEMzevOBUi1Yo0wHzN/g== 2020-09-08T17:04:59.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC7uVdfLlbVDRAAW7ZEMzevOBUi1Yo0wHzN/g=="}' 2020-09-08T17:04:59.226 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ea876442-f5ea-4950-a3b2-b9feb220b85c -i td/test-erasure-eio/1/new.json 2020-09-08T17:04:59.752 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:04:59.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:04:59.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC7uVdfLlbVDRAAW7ZEMzevOBUi1Yo0wHzN/g== --osd-uuid ea876442-f5ea-4950-a3b2-b9feb220b85c 2020-09-08T17:04:59.784 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:59.789+0000 7f2c2bd70ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:59.786 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:59.791+0000 7f2c2bd70ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:59.787 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:59.793+0000 7f2c2bd70ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:04:59.788 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:59.793+0000 7f2c2bd70ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:04:59.788 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:59.793+0000 7f2c2bd70ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:04:59.788 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:59.793+0000 7f2c2bd70ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:04:59.828 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:04:59.833+0000 7f2c2bd70ec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:05:03.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:05:03.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:05:03.794 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:05:03.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:05:03.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:05:04.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:05:04.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:04.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:05:04.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:05:04.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:05:04.296 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:05:04.313 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:04.319+0000 7f23a922eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:04.335 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:04.340+0000 7f23a922eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:04.337 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:04.342+0000 7f23a922eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:04.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:05:04.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:05:04.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:05:04.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:05:04.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:05:04.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:04.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:05:04.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:04.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:05:04.663 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:05:04.852 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:04.858+0000 7f23a922eec0 -1 Falling back to public interface 2020-09-08T17:05:05.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:05.636 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:05.641+0000 7f23a922eec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:05:06.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:06.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:06.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:05:06.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:06.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:05:06.033 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:05:06.472 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/131978,v1:127.0.0.1:6811/131978] [v2:127.0.0.1:6812/131978,v1:127.0.0.1:6813/131978] exists,up ea876442-f5ea-4950-a3b2-b9feb220b85c 2020-09-08T17:05:06.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:05:06.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:05:06.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:05:06.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:05:06.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:05:06.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:05:06.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:05:06.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:05:06.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:05:06.474 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:05:06.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:05:06.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:05:06.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:05:06.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:05:06.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:05:06.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:05:06.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:05:06.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:05:06.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:05:06.476 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:05:06.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:05:06.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:05:06.477 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:05:06.477 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:06.477 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:06.478 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:06.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:06.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:05:06.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:05:06.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:05:06.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:05:06.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:05:06.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:05:06.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:05:06.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:06.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:05:06.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:05:06.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:05:06.480 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:05:06.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a32c291a-d29c-472f-b953-0b48347860ce 2020-09-08T17:05:06.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 a32c291a-d29c-472f-b953-0b48347860ce' 2020-09-08T17:05:06.481 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:05:06.482 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 a32c291a-d29c-472f-b953-0b48347860ce 2020-09-08T17:05:06.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDCuVdfHsJNHRAA3SLIF3tFK2zysnLJ/8zSJw== 2020-09-08T17:05:06.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDCuVdfHsJNHRAA3SLIF3tFK2zysnLJ/8zSJw=="}' 2020-09-08T17:05:06.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a32c291a-d29c-472f-b953-0b48347860ce -i td/test-erasure-eio/2/new.json 2020-09-08T17:05:07.195 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:05:07.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:05:07.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDCuVdfHsJNHRAA3SLIF3tFK2zysnLJ/8zSJw== --osd-uuid a32c291a-d29c-472f-b953-0b48347860ce 2020-09-08T17:05:07.226 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:07.232+0000 7efde9f80ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:07.228 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:07.233+0000 7efde9f80ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:07.229 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:07.235+0000 7efde9f80ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:07.230 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:07.236+0000 7efde9f80ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:05:07.230 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:07.236+0000 7efde9f80ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:05:07.231 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:07.236+0000 7efde9f80ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:05:07.306 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:07.311+0000 7efde9f80ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:05:11.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:05:11.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:05:11.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:05:11.465 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:05:11.465 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:05:11.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:05:11.961 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:05:11.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:05:11.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:11.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:05:11.963 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:05:11.980 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:11.985+0000 7f43078f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:11.981 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:11.987+0000 7f43078f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:11.983 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:11.988+0000 7f43078f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:12.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:05:12.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:05:12.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:05:12.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:05:12.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:05:12.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:12.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:05:12.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:12.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:05:12.330 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:05:12.498 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:12.503+0000 7f43078f4ec0 -1 Falling back to public interface 2020-09-08T17:05:12.690 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:13.305 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:13.310+0000 7f43078f4ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:05:13.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:13.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:13.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:05:13.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:13.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:05:13.693 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:05:14.080 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/132974,v1:127.0.0.1:6819/132974] [v2:127.0.0.1:6820/132974,v1:127.0.0.1:6821/132974] exists,up a32c291a-d29c-472f-b953-0b48347860ce 2020-09-08T17:05:14.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:05:14.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:05:14.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:05:14.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:05:14.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:05:14.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:05:14.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:05:14.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:05:14.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:05:14.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:05:14.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:05:14.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:05:14.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:05:14.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:05:14.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:05:14.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:05:14.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:05:14.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:05:14.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:05:14.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:05:14.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:05:14.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:05:14.085 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:05:14.085 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:14.086 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:14.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:14.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:14.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:05:14.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:05:14.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:05:14.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:05:14.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:05:14.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:05:14.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:05:14.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:14.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:05:14.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:05:14.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:05:14.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:05:14.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f8a99cd1-84d1-4dfb-87d3-4f94742dbce8 2020-09-08T17:05:14.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 f8a99cd1-84d1-4dfb-87d3-4f94742dbce8' 2020-09-08T17:05:14.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:05:14.090 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 f8a99cd1-84d1-4dfb-87d3-4f94742dbce8 2020-09-08T17:05:14.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDKuVdftkPzBRAA+AqkeVPIIYV729LfvfwMbw== 2020-09-08T17:05:14.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDKuVdftkPzBRAA+AqkeVPIIYV729LfvfwMbw=="}' 2020-09-08T17:05:14.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f8a99cd1-84d1-4dfb-87d3-4f94742dbce8 -i td/test-erasure-eio/3/new.json 2020-09-08T17:05:14.561 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:05:14.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:05:14.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKuVdftkPzBRAA+AqkeVPIIYV729LfvfwMbw== --osd-uuid f8a99cd1-84d1-4dfb-87d3-4f94742dbce8 2020-09-08T17:05:14.593 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:14.599+0000 7f778c695ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:14.605 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:14.610+0000 7f778c695ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:14.605 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:14.611+0000 7f778c695ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:14.606 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:14.611+0000 7f778c695ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:05:14.606 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:14.611+0000 7f778c695ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:05:14.607 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:14.611+0000 7f778c695ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:05:14.693 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:14.699+0000 7f778c695ec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:05:19.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:05:19.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:05:19.671 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:05:19.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:05:19.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:05:20.172 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:05:20.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:05:20.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:20.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:05:20.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:05:20.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:05:20.191 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:20.196+0000 7ff64af2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:20.193 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:20.198+0000 7ff64af2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:20.194 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:20.200+0000 7ff64af2dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:20.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:05:20.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:05:20.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:05:20.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:05:20.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:05:20.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:20.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:05:20.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:20.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:05:20.544 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:05:20.710 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:20.715+0000 7ff64af2dec0 -1 Falling back to public interface 2020-09-08T17:05:20.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:21.512 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:21.517+0000 7ff64af2dec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:05:21.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:21.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:21.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:05:21.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:21.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:05:21.909 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:05:22.391 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/133968,v1:127.0.0.1:6827/133968] [v2:127.0.0.1:6828/133968,v1:127.0.0.1:6829/133968] exists,up f8a99cd1-84d1-4dfb-87d3-4f94742dbce8 2020-09-08T17:05:22.391 INFO:tasks.workunit.client.0.smithi120.stdout:add osd4 b2523e23-41b9-4091-b9c3-40154873fe56 2020-09-08T17:05:22.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:05:22.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:05:22.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:05:22.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:05:22.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 4 2020-09-08T17:05:22.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:05:22.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:05:22.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2020-09-08T17:05:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:05:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/4 2020-09-08T17:05:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:05:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:05:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:05:22.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:05:22.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2020-09-08T17:05:22.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2020-09-08T17:05:22.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:05:22.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:05:22.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:05:22.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:05:22.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:05:22.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:05:22.396 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:05:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:22.397 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:22.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:05:22.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:05:22.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:05:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:05:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:05:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:05:22.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:05:22.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:22.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:05:22.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:05:22.410 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/4 2020-09-08T17:05:22.410 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:05:22.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b2523e23-41b9-4091-b9c3-40154873fe56 2020-09-08T17:05:22.411 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 b2523e23-41b9-4091-b9c3-40154873fe56' 2020-09-08T17:05:22.411 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:05:22.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDSuVdfJWJhGBAAolGc02j/tFawipEu9bnoUA== 2020-09-08T17:05:22.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDSuVdfJWJhGBAAolGc02j/tFawipEu9bnoUA=="}' 2020-09-08T17:05:22.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b2523e23-41b9-4091-b9c3-40154873fe56 -i td/test-erasure-eio/4/new.json 2020-09-08T17:05:23.167 INFO:tasks.workunit.client.0.smithi120.stdout:4 2020-09-08T17:05:23.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/4/new.json 2020-09-08T17:05:23.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDSuVdfJWJhGBAAolGc02j/tFawipEu9bnoUA== --osd-uuid b2523e23-41b9-4091-b9c3-40154873fe56 2020-09-08T17:05:23.199 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:23.204+0000 7fae0cca8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:23.225 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:23.230+0000 7fae0cca8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:23.227 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:23.231+0000 7fae0cca8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:23.227 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:23.232+0000 7fae0cca8ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:05:23.227 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:23.232+0000 7fae0cca8ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:05:23.228 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:23.232+0000 7fae0cca8ec0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2020-09-08T17:05:23.334 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:23.340+0000 7fae0cca8ec0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2020-09-08T17:05:28.040 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2020-09-08T17:05:28.040 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:05:28.041 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd4 key to auth repository 2020-09-08T17:05:28.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2020-09-08T17:05:28.041 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:05:28.542 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.4 2020-09-08T17:05:28.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2020-09-08T17:05:28.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:28.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:05:28.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:05:28.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:05:28.561 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:28.567+0000 7f6e258caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:28.563 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:28.569+0000 7f6e258caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:28.563 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:28.570+0000 7f6e258caec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:28.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2020-09-08T17:05:28.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:05:28.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2020-09-08T17:05:28.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:05:28.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:05:28.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:28.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:05:28.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:28.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:05:28.933 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:05:29.079 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:29.085+0000 7f6e258caec0 -1 Falling back to public interface 2020-09-08T17:05:29.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:30.087 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:30.094+0000 7f6e258caec0 -1 osd.4 0 log_to_monitors {default=true} 2020-09-08T17:05:30.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:30.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:30.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:05:30.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:30.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:05:30.300 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:05:30.703 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:31.704 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:31.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:31.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:05:31.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:31.705 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2020-09-08T17:05:31.706 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:05:32.138 INFO:tasks.workunit.client.0.smithi120.stdout:osd.4 up in weight 1 up_from 32 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/134962,v1:127.0.0.1:6835/134962] [v2:127.0.0.1:6836/134962,v1:127.0.0.1:6837/134962] exists,up b2523e23-41b9-4091-b9c3-40154873fe56 2020-09-08T17:05:32.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:05:32.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:05:32.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:05:32.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:05:32.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 5 2020-09-08T17:05:32.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:05:32.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:05:32.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2020-09-08T17:05:32.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:05:32.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/5 2020-09-08T17:05:32.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:05:32.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:05:32.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:05:32.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:05:32.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2020-09-08T17:05:32.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2020-09-08T17:05:32.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:05:32.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:05:32.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:05:32.142 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:05:32.142 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:05:32.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:05:32.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:05:32.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:32.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:32.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:32.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:32.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:05:32.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:05:32.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:05:32.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:05:32.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:05:32.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:05:32.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:05:32.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:32.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:05:32.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:05:32.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/5 2020-09-08T17:05:32.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:05:32.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a4ed5428-296f-4eab-8165-59793a79f794 2020-09-08T17:05:32.147 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 a4ed5428-296f-4eab-8165-59793a79f794' 2020-09-08T17:05:32.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:05:32.148 INFO:tasks.workunit.client.0.smithi120.stdout:add osd5 a4ed5428-296f-4eab-8165-59793a79f794 2020-09-08T17:05:32.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDcuVdfygZsCRAABIlEJtNB3Ffxkzj9Z4gBxQ== 2020-09-08T17:05:32.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDcuVdfygZsCRAABIlEJtNB3Ffxkzj9Z4gBxQ=="}' 2020-09-08T17:05:32.152 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a4ed5428-296f-4eab-8165-59793a79f794 -i td/test-erasure-eio/5/new.json 2020-09-08T17:05:32.638 INFO:tasks.workunit.client.0.smithi120.stdout:5 2020-09-08T17:05:32.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/5/new.json 2020-09-08T17:05:32.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDcuVdfygZsCRAABIlEJtNB3Ffxkzj9Z4gBxQ== --osd-uuid a4ed5428-296f-4eab-8165-59793a79f794 2020-09-08T17:05:32.669 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:32.675+0000 7faeb46c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:32.670 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:32.677+0000 7faeb46c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:32.673 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:32.679+0000 7faeb46c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:32.673 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:32.679+0000 7faeb46c9ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:05:32.673 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:32.679+0000 7faeb46c9ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:05:32.674 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:32.679+0000 7faeb46c9ec0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2020-09-08T17:05:32.791 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:32.797+0000 7faeb46c9ec0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2020-09-08T17:05:37.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2020-09-08T17:05:37.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:05:37.034 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd5 key to auth repository 2020-09-08T17:05:37.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2020-09-08T17:05:37.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:05:37.528 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.5 2020-09-08T17:05:37.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2020-09-08T17:05:37.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:05:37.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:05:37.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:05:37.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:37.546 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:37.552+0000 7f5cb80afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:37.548 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:37.554+0000 7f5cb80afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:37.549 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:37.556+0000 7f5cb80afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:37.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2020-09-08T17:05:37.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:05:37.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2020-09-08T17:05:37.893 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:05:37.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:05:37.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:37.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:05:37.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:37.894 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:05:37.895 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:05:38.066 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:38.072+0000 7f5cb80afec0 -1 Falling back to public interface 2020-09-08T17:05:38.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:38.921 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:38.926+0000 7f5cb80afec0 -1 osd.5 0 log_to_monitors {default=true} 2020-09-08T17:05:39.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:39.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:39.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:05:39.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:39.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:05:39.265 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:05:39.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:40.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:40.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:40.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:05:40.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:40.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2020-09-08T17:05:40.634 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:05:41.072 INFO:tasks.workunit.client.0.smithi120.stdout:osd.5 up in weight 1 up_from 40 up_thru 41 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/135990,v1:127.0.0.1:6843/135990] [v2:127.0.0.1:6844/135990,v1:127.0.0.1:6845/135990] exists,up a4ed5428-296f-4eab-8165-59793a79f794 2020-09-08T17:05:41.072 INFO:tasks.workunit.client.0.smithi120.stdout:add osd6 d4b416c5-df7a-42d8-a7bc-c4c09a68836d 2020-09-08T17:05:41.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:05:41.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:05:41.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:05:41.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:05:41.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 6 2020-09-08T17:05:41.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:05:41.074 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:05:41.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2020-09-08T17:05:41.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:05:41.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/6 2020-09-08T17:05:41.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:05:41.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:05:41.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:05:41.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:05:41.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2020-09-08T17:05:41.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2020-09-08T17:05:41.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:05:41.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:05:41.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:05:41.077 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:05:41.077 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:05:41.078 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:05:41.078 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:05:41.079 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:41.079 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:41.079 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:41.079 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:05:41.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:05:41.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:05:41.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:05:41.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:05:41.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:05:41.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:05:41.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:05:41.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:41.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:05:41.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:05:41.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/6 2020-09-08T17:05:41.082 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:05:41.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d4b416c5-df7a-42d8-a7bc-c4c09a68836d 2020-09-08T17:05:41.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 d4b416c5-df7a-42d8-a7bc-c4c09a68836d' 2020-09-08T17:05:41.082 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:05:41.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDluVdfvfpSBRAAfCVMZTGio2rCD5lEzznqFw== 2020-09-08T17:05:41.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDluVdfvfpSBRAAfCVMZTGio2rCD5lEzznqFw=="}' 2020-09-08T17:05:41.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d4b416c5-df7a-42d8-a7bc-c4c09a68836d -i td/test-erasure-eio/6/new.json 2020-09-08T17:05:41.583 INFO:tasks.workunit.client.0.smithi120.stdout:6 2020-09-08T17:05:41.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/6/new.json 2020-09-08T17:05:41.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDluVdfvfpSBRAAfCVMZTGio2rCD5lEzznqFw== --osd-uuid d4b416c5-df7a-42d8-a7bc-c4c09a68836d 2020-09-08T17:05:41.615 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:41.620+0000 7faf49a22ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:41.616 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:41.622+0000 7faf49a22ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:41.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:41.623+0000 7faf49a22ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:41.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:41.624+0000 7faf49a22ec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:05:41.619 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:41.624+0000 7faf49a22ec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:05:41.619 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:41.624+0000 7faf49a22ec0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2020-09-08T17:05:41.748 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:41.754+0000 7faf49a22ec0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2020-09-08T17:05:45.960 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2020-09-08T17:05:45.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:05:45.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2020-09-08T17:05:45.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:05:45.962 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd6 key to auth repository 2020-09-08T17:05:46.469 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.6 2020-09-08T17:05:46.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2020-09-08T17:05:46.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:05:46.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:05:46.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:05:46.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:05:46.488 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:46.493+0000 7ff74076cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:46.490 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:46.495+0000 7ff74076cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:46.491 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:46.496+0000 7ff74076cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:05:46.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2020-09-08T17:05:46.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:05:46.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2020-09-08T17:05:46.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:05:46.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:05:46.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:46.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:05:46.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:46.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:05:46.841 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:05:47.014 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:47.020+0000 7ff74076cec0 -1 Falling back to public interface 2020-09-08T17:05:47.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:05:47.777 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:05:47.782+0000 7ff74076cec0 -1 osd.6 0 log_to_monitors {default=true} 2020-09-08T17:05:48.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:05:48.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:05:48.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:05:48.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:05:48.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2020-09-08T17:05:48.206 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:05:48.696 INFO:tasks.workunit.client.0.smithi120.stdout:osd.6 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/137010,v1:127.0.0.1:6851/137010] [v2:127.0.0.1:6852/137010,v1:127.0.0.1:6853/137010] exists,up d4b416c5-df7a-42d8-a7bc-c4c09a68836d 2020-09-08T17:05:48.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:05:48.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:05:48.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:05:48.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:05:48.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:05:48.698 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:05:48.698 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:05:48.698 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:48.699 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:48.699 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:05:48.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:05:48.699 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:05:48.789 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:05:48.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:05:48.799 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:04:57.544+0000 7f9646e39ec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:05:48.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:394: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2020-09-08T17:05:48.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2020-09-08T17:05:48.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:05:48.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:05:48.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=3 2020-09-08T17:05:48.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:05:48.800 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=2 2020-09-08T17:05:48.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:05:48.801 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2020-09-08T17:05:49.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:05:49.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:05:50.197 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:05:50.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:05:51.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:05:51.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:05:51.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:05:51.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:05:51.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:05:51.214 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:05:51.214 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:05:51.214 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:05:51.214 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:05:51.214 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:05:51.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:05:51.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:05:51.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:05:51.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:05:51.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:05:51.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:05:51.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:05:51.707 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:05:51.707 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:05:51.707 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:05:51.707 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:05:51.707 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:05:51.708 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:05:51.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:05:51.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:51.708 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:05:51.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803789 2020-09-08T17:05:51.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803789 2020-09-08T17:05:51.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789' 2020-09-08T17:05:51.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:51.828 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:05:51.948 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607564 2020-09-08T17:05:51.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607564 2020-09-08T17:05:51.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607564' 2020-09-08T17:05:51.949 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:51.950 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:05:52.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411338 2020-09-08T17:05:52.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411338 2020-09-08T17:05:52.069 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-77309411338' 2020-09-08T17:05:52.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:52.070 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:05:52.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149704 2020-09-08T17:05:52.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149704 2020-09-08T17:05:52.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-77309411338 3-111669149704' 2020-09-08T17:05:52.190 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:52.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:05:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=137438953480 2020-09-08T17:05:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 137438953480 2020-09-08T17:05:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-77309411338 3-111669149704 4-137438953480' 2020-09-08T17:05:52.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:52.349 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T17:05:52.467 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691846 2020-09-08T17:05:52.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691846 2020-09-08T17:05:52.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-77309411338 3-111669149704 4-137438953480 5-171798691846' 2020-09-08T17:05:52.468 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:52.468 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T17:05:52.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495619 2020-09-08T17:05:52.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495619 2020-09-08T17:05:52.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607564 2-77309411338 3-111669149704 4-137438953480 5-171798691846 6-197568495619' 2020-09-08T17:05:52.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:05:52.588 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803789 2020-09-08T17:05:52.588 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:05:52.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:05:52.589 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803789 2020-09-08T17:05:52.589 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:05:52.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803789 2020-09-08T17:05:52.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2020-09-08T17:05:52.590 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:05:52.591 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803789 2020-09-08T17:05:53.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803789 -lt 25769803789 2020-09-08T17:05:53.001 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:05:53.002 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607564 2020-09-08T17:05:53.002 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:05:53.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:05:53.003 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607564 2020-09-08T17:05:53.003 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:05:53.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607564 2020-09-08T17:05:53.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607564' 2020-09-08T17:05:53.004 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:05:53.005 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 51539607564 2020-09-08T17:05:53.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607564 -lt 51539607564 2020-09-08T17:05:53.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:05:53.370 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411338 2020-09-08T17:05:53.371 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:05:53.371 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:05:53.371 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411338 2020-09-08T17:05:53.372 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:05:53.372 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411338 2020-09-08T17:05:53.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411338' 2020-09-08T17:05:53.373 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:05:53.373 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411338 2020-09-08T17:05:53.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411338 -lt 77309411338 2020-09-08T17:05:53.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:05:53.738 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149704 2020-09-08T17:05:53.739 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:05:53.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:05:53.739 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149704 2020-09-08T17:05:53.740 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:05:53.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149704 2020-09-08T17:05:53.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149704' 2020-09-08T17:05:53.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:05:53.741 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149704 2020-09-08T17:05:54.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149704 -lt 111669149704 2020-09-08T17:05:54.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:05:54.120 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-137438953480 2020-09-08T17:05:54.121 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:05:54.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:05:54.122 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-137438953480 2020-09-08T17:05:54.122 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:05:54.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=137438953480 2020-09-08T17:05:54.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 137438953480' 2020-09-08T17:05:54.123 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:05:54.123 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 137438953480 2020-09-08T17:05:54.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 137438953480 -lt 137438953480 2020-09-08T17:05:54.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:05:54.530 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-171798691846 2020-09-08T17:05:54.530 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:05:54.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T17:05:54.531 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-171798691846 2020-09-08T17:05:54.532 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:05:54.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691846 2020-09-08T17:05:54.532 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 171798691846' 2020-09-08T17:05:54.532 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 171798691846 2020-09-08T17:05:54.533 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T17:05:54.905 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691846 -lt 171798691846 2020-09-08T17:05:54.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:05:54.906 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-197568495619 2020-09-08T17:05:54.906 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:05:54.907 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2020-09-08T17:05:54.907 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-197568495619 2020-09-08T17:05:54.907 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:05:54.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495619 2020-09-08T17:05:54.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 197568495619' 2020-09-08T17:05:54.908 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2020-09-08T17:05:54.909 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.6 seq 197568495619 2020-09-08T17:05:55.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495620 -lt 197568495619 2020-09-08T17:05:55.272 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:05:55.272 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:05:55.272 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:05:55.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:05:55.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:05:55.759 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:05:55.759 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:05:55.759 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:05:55.760 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:05:55.760 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:05:55.760 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:05:56.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:05:56.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:05:56.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:05:56.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:05:56.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:05:56.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:05:56.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:05:56.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:397: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2020-09-08T17:05:56.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:05:56.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:05:56.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=myobject 2020-09-08T17:05:56.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:05:56.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:05:56.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:05:56.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:05:56.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:05:56.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:05:56.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:05:56.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:05:56.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2020-09-08T17:05:56.751 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2020-09-08T17:05:56.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:05:56.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:05:56.752 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:05:56.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:05:56.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:05:56.753 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:05:56.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=myobject 2020-09-08T17:05:56.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:05:56.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:05:56.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:05:56.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2020-09-08T17:05:56.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:05:56.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:05:56.755 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure myobject 2020-09-08T17:05:56.755 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:05:56.755 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T17:05:56.756 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T17:05:56.756 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:05:57.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:05:57.146 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:05:57.147 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:05:57.147 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:05:57.147 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:05:57.147 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 1 0 6 2020-09-08T17:05:57.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:05:57.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2020-09-08T17:05:57.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:05:57.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/3/type 2020-09-08T17:05:57.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:05:57.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2020-09-08T17:05:57.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:05:57.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2020-09-08T17:05:57.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:05:57.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:05:57.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:05:57.151 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2020-09-08T17:05:57.151 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:05:57.152 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:05:57.152 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:05:57.152 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:57.152 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:57.152 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:05:57.153 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:05:57.242 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:05:57.243 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:05:57.244 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:05:57.244 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2020-09-08T17:05:57.244 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:05:57.245 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:05:57.245 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:05:57.245 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:05:57.245 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:05:57.246 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:05:57.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:05:57.246 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2020-09-08T17:05:57.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:400: TEST_ec_single_recovery_error: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:05:57.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:400: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2020-09-08T17:05:57.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:05:57.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2020-09-08T17:05:57.341 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure myobject 2020-09-08T17:05:57.341 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:05:57.707 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:05:57.708 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:05:57.708 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:05:57.708 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:05:57.708 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:05:57.708 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 5 1 0 6 2020-09-08T17:05:57.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:400: TEST_ec_single_recovery_error: local -a initial_osds 2020-09-08T17:05:57.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local last_osd=6 2020-09-08T17:05:57.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:403: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2020-09-08T17:05:57.709 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:05:57.710 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:05:57.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:05:57.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:05:57.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:05:57.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:05:57.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: ceph osd down 6 2020-09-08T17:05:58.199 INFO:tasks.workunit.client.0.smithi120.stderr:osd.6 is already down. 2020-09-08T17:05:58.207 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd out 6 2020-09-08T17:05:58.903 INFO:tasks.workunit.client.0.smithi120.stderr:osd.6 is already out. 2020-09-08T17:05:58.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:408: TEST_ec_single_recovery_error: wait_for_clean 2020-09-08T17:05:58.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:05:58.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:05:58.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:05:58.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:05:58.913 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:05:58.914 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:05:58.914 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:05:58.914 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:05:58.914 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:05:58.914 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:05:59.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:05:59.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:05:59.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:05:59.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:05:59.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:05:59.373 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:05:59.373 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:05:59.374 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:05:59.374 INFO:tasks.workunit.client.0.smithi120.stderr:3 2020-09-08T17:05:59.379 INFO:tasks.workunit.client.0.smithi120.stderr:4 2020-09-08T17:05:59.379 INFO:tasks.workunit.client.0.smithi120.stderr:5 2020-09-08T17:05:59.379 INFO:tasks.workunit.client.0.smithi120.stderr:6' 2020-09-08T17:05:59.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:05:59.380 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:59.380 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:05:59.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803792 2020-09-08T17:05:59.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803792 2020-09-08T17:05:59.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792' 2020-09-08T17:05:59.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:59.564 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:05:59.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607567 2020-09-08T17:05:59.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607567 2020-09-08T17:05:59.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-51539607567' 2020-09-08T17:05:59.686 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:59.687 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:05:59.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411342 2020-09-08T17:05:59.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411342 2020-09-08T17:05:59.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-77309411342' 2020-09-08T17:05:59.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:59.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:05:59.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149708 2020-09-08T17:05:59.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149708 2020-09-08T17:05:59.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-77309411342 3-111669149708' 2020-09-08T17:05:59.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:05:59.928 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-09-08T17:06:00.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=137438953483 2020-09-08T17:06:00.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 137438953483 2020-09-08T17:06:00.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-77309411342 3-111669149708 4-137438953483' 2020-09-08T17:06:00.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:00.050 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-09-08T17:06:00.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691849 2020-09-08T17:06:00.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691849 2020-09-08T17:06:00.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-51539607567 2-77309411342 3-111669149708 4-137438953483 5-171798691849' 2020-09-08T17:06:00.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:00.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2020-09-08T17:06:00.327 INFO:tasks.workunit.client.0.smithi120.stderr:Error ENXIO: problem getting command descriptions from osd.6 2020-09-08T17:06:00.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2020-09-08T17:06:00.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2020-09-08T17:06:00.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2020-09-08T17:06:00.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:00.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803792 2020-09-08T17:06:00.329 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:00.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:06:00.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803792 2020-09-08T17:06:00.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:00.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803792 2020-09-08T17:06:00.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803792' 2020-09-08T17:06:00.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:06:00.332 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803792 2020-09-08T17:06:00.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803793 -lt 25769803792 2020-09-08T17:06:00.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:00.823 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607567 2020-09-08T17:06:00.823 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:00.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:06:00.824 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607567 2020-09-08T17:06:00.824 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:00.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607567 2020-09-08T17:06:00.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607567' 2020-09-08T17:06:00.825 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:06:00.825 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 51539607567 2020-09-08T17:06:01.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607567 -lt 51539607567 2020-09-08T17:06:01.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:01.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411342 2020-09-08T17:06:01.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:01.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:06:01.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411342 2020-09-08T17:06:01.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:01.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411342 2020-09-08T17:06:01.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411342' 2020-09-08T17:06:01.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:06:01.190 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411342 2020-09-08T17:06:01.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411342 -lt 77309411342 2020-09-08T17:06:01.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:01.558 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149708 2020-09-08T17:06:01.559 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:01.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:06:01.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149708 2020-09-08T17:06:01.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:01.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149708 2020-09-08T17:06:01.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149708' 2020-09-08T17:06:01.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:06:01.562 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149708 2020-09-08T17:06:01.953 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149708 -lt 111669149708 2020-09-08T17:06:01.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:01.954 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-137438953483 2020-09-08T17:06:01.954 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:01.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2020-09-08T17:06:01.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-137438953483 2020-09-08T17:06:01.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:01.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=137438953483 2020-09-08T17:06:01.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 137438953483' 2020-09-08T17:06:01.957 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2020-09-08T17:06:01.957 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.4 seq 137438953483 2020-09-08T17:06:02.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 137438953483 -lt 137438953483 2020-09-08T17:06:02.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:02.321 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-171798691849 2020-09-08T17:06:02.321 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:02.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2020-09-08T17:06:02.322 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-171798691849 2020-09-08T17:06:02.322 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:02.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691849 2020-09-08T17:06:02.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 171798691849' 2020-09-08T17:06:02.323 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2020-09-08T17:06:02.324 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.5 seq 171798691849 2020-09-08T17:06:02.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691850 -lt 171798691849 2020-09-08T17:06:02.688 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:06:02.689 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:06:02.689 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:06:03.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:06:03.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:06:03.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:06:03.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:06:03.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:06:03.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:06:03.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:06:03.174 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:06:03.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-09-08T17:06:03.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:06:03.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:06:03.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:06:04.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 5 2020-09-08T17:06:04.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 '!=' -1 2020-09-08T17:06:04.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T17:06:04.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=3 2020-09-08T17:06:04.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:06:04.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T17:06:04.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:06:04.104 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:06:04.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:06:04.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:06:04.104 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:06:04.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:06:04.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:06:04.105 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:06:04.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T17:06:04.465 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:06:04.465 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:06:04.466 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:06:04.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T17:06:04.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 3 2020-09-08T17:06:04.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T17:06:04.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=4 2020-09-08T17:06:04.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:06:04.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T17:06:05.045 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:06:05.046 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:06:05.046 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:06:05.046 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:06:05.046 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:06:05.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:06:05.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:06:05.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:06:05.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T17:06:05.397 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:06:05.398 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:06:05.398 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:06:05.881 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T17:06:05.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T17:06:05.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T17:06:05.882 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T17:06:05.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T17:06:05.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T17:06:05.883 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T17:06:05.883 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T17:06:05.883 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T17:06:06.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=550 2020-09-08T17:06:06.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 550 '!=' null 2020-09-08T17:06:06.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2020-09-08T17:06:06.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:06:06.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T17:06:06.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:06:06.464 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:06:06.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:06:06.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:06:06.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:06:06.465 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:06:06.465 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:06:06.465 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:06:06.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:06:06.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:06:06.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:06:06.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:06:07.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:06:07.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:06:07.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:06:07.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:410: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2020-09-08T17:06:07.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:06:07.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:06:07.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=myobject 2020-09-08T17:06:07.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:06:07.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:06:07.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2020-09-08T17:06:07.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:06:07.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:06:07.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:412: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:06:07.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:06:07.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:06:07.851 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:06:07.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:06:08.419 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:06:08.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:06:08.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:06:08.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:06:08.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:06:08.429 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:06:08.429 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:06:08.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:06:08.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:06:08.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:06:08.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:06:08.538 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:06:08.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:06:08.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:06:08.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:06:08.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:06:08.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:06:08.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:06:08.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:06:08.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:06:08.543 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:06:08.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:06:08.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:06:08.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:06:08.553 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:06:08.553 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:08.553 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:08.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:06:08.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:06:08.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:06:08.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:06:08.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:06:08.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:06:08.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:06:08.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:06:08.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:06:08.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:06:08.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:06:08.559 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:06:08.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:06:08.559 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:06:08.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:06:08.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:06:08.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:06:08.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:06:08.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:06:08.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:06:08.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:06:08.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:06:08.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:06:08.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:06:08.563 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:06:08.563 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:08.564 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:08.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:06:08.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:06:08.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:06:08.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:06:08.566 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:06:08.566 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:08.566 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:08.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:06:08.568 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:06:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:06:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:06:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:06:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:06:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:06:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:06:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:06:08.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:06:08.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:06:08.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:06:08.857 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:06:08.857 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:06:08.858 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:06:08.858 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:06:08.858 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:08.858 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:08.858 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:08.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:06:09.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:06:09.026 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:06:09.027 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:06:09.027 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:06:09.027 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:06:09.027 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:06:09.027 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:06:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:06:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:06:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:06:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:09.029 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:06:09.029 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:06:09.029 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:06:09.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:06:09.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:06:09.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:06:09.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:06:09.115 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:06:09.115 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:06:09.115 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:06:09.116 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:06:09.116 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:06:09.116 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:09.116 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:09.116 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:06:09.117 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:06:09.117 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:06:09.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:06:09.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:06:09.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:06:09.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:06:09.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:06:09.202 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:06:09.203 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:06:10.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:06:10.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:06:10.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:06:10.085 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:06:10.086 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:10.086 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:10.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:10.087 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:06:10.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:06:10.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:06:10.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:06:10.393 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:06:10.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:06:11.403 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:06:11.404 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:06:11.404 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:06:11.404 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:06:11.404 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:11.404 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:11.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:06:11.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:06:11.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:06:11.484 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:06:11.493 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:06:11.494 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:06:08.891+0000 7f4dd75736c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:06:11.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2020-09-08T17:06:11.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:301: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2020-09-08T17:06:11.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: setup_osds 4 2020-09-08T17:06:11.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=4 2020-09-08T17:06:11.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:06:11.496 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 4 - 1 2020-09-08T17:06:11.496 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 3 2020-09-08T17:06:11.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:06:11.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:06:11.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:06:11.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:06:11.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:06:11.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:06:11.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:06:11.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:06:11.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:06:11.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:06:11.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:06:11.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:06:11.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:06:11.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:06:11.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:06:11.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:06:11.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:06:11.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:06:11.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:06:11.505 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:06:11.505 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:11.505 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:11.506 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:11.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:11.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:06:11.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:06:11.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:06:11.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:06:11.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:06:11.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:06:11.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:06:11.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:11.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:06:11.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:06:11.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:06:11.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:06:11.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cc0dbea6-e280-4611-aee0-4c43ed758943 2020-09-08T17:06:11.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 cc0dbea6-e280-4611-aee0-4c43ed758943' 2020-09-08T17:06:11.509 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:06:11.509 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 cc0dbea6-e280-4611-aee0-4c43ed758943 2020-09-08T17:06:11.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQADuldf/lP5HhAAM+hCqnZIW0LSyLD4IVfIEQ== 2020-09-08T17:06:11.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQADuldf/lP5HhAAM+hCqnZIW0LSyLD4IVfIEQ=="}' 2020-09-08T17:06:11.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cc0dbea6-e280-4611-aee0-4c43ed758943 -i td/test-erasure-eio/0/new.json 2020-09-08T17:06:11.834 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:06:11.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:06:11.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQADuldf/lP5HhAAM+hCqnZIW0LSyLD4IVfIEQ== --osd-uuid cc0dbea6-e280-4611-aee0-4c43ed758943 2020-09-08T17:06:11.866 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:11.872+0000 7ff2cf16cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:11.868 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:11.874+0000 7ff2cf16cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:11.869 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:11.875+0000 7ff2cf16cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:11.869 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:11.876+0000 7ff2cf16cec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:06:11.870 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:11.876+0000 7ff2cf16cec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:06:11.870 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:11.876+0000 7ff2cf16cec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:06:11.934 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:11.940+0000 7ff2cf16cec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:06:15.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:06:15.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:06:15.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:06:15.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:06:15.861 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:06:16.354 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:06:16.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:06:16.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:16.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:06:16.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:06:16.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:06:16.373 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:16.380+0000 7ff2a93e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:16.375 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:16.381+0000 7ff2a93e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:16.376 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:16.382+0000 7ff2a93e1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:16.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:06:16.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:06:16.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:06:16.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:06:16.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:06:16.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:16.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:06:16.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:16.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:06:16.722 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:06:16.897 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:16.903+0000 7ff2a93e1ec0 -1 Falling back to public interface 2020-09-08T17:06:17.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:06:17.666 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:17.673+0000 7ff2a93e1ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:06:18.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:06:18.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:18.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:06:18.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:18.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:06:18.086 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:06:18.469 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/139922,v1:127.0.0.1:6803/139922] [v2:127.0.0.1:6804/139922,v1:127.0.0.1:6805/139922] exists,up cc0dbea6-e280-4611-aee0-4c43ed758943 2020-09-08T17:06:18.469 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:06:18.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:06:18.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:06:18.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:06:18.470 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:06:18.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:06:18.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:06:18.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:06:18.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:06:18.471 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:06:18.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:06:18.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:06:18.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:06:18.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:06:18.472 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:06:18.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:06:18.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:06:18.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:06:18.473 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:06:18.473 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:06:18.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:06:18.474 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:06:18.474 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:06:18.474 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:18.474 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:18.475 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:18.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:18.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:06:18.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:06:18.475 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:06:18.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:06:18.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:06:18.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:06:18.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:06:18.476 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:18.477 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:06:18.477 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:06:18.477 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:06:18.477 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:06:18.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:06:18.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 285af751-9c8b-4712-a97b-8e52a98e090e' 2020-09-08T17:06:18.479 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:06:18.479 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:06:18.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAKuldfoWooHRAALEv2sODpgc7n8Q01FEVL8w== 2020-09-08T17:06:18.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAKuldfoWooHRAALEv2sODpgc7n8Q01FEVL8w=="}' 2020-09-08T17:06:18.483 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 285af751-9c8b-4712-a97b-8e52a98e090e -i td/test-erasure-eio/1/new.json 2020-09-08T17:06:19.092 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:06:19.102 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:06:19.103 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAKuldfoWooHRAALEv2sODpgc7n8Q01FEVL8w== --osd-uuid 285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:06:19.124 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:19.130+0000 7ff25fcafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:19.125 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:19.132+0000 7ff25fcafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:19.126 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:19.133+0000 7ff25fcafec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:19.127 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:19.133+0000 7ff25fcafec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:06:19.127 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:19.133+0000 7ff25fcafec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:06:19.127 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:19.133+0000 7ff25fcafec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:06:19.169 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:19.174+0000 7ff25fcafec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:06:23.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:06:23.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:06:23.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:06:23.110 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:06:23.111 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:06:23.607 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:06:23.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:06:23.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:23.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:06:23.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:06:23.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:06:23.625 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:23.630+0000 7f67f50c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:23.626 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:23.632+0000 7f67f50c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:23.628 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:23.633+0000 7f67f50c9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:23.973 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:06:23.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:06:23.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:06:23.974 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:06:23.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:06:23.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:23.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:06:23.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:23.975 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:06:23.976 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:06:24.143 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:24.148+0000 7f67f50c9ec0 -1 Falling back to public interface 2020-09-08T17:06:24.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:06:24.918 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:24.923+0000 7f67f50c9ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:06:25.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:06:25.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:25.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:06:25.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:25.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:06:25.377 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:06:25.754 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/140916,v1:127.0.0.1:6811/140916] [v2:127.0.0.1:6812/140916,v1:127.0.0.1:6813/140916] exists,up 285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:06:25.755 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 71530084-7918-4ff2-b657-4c0fab058757 2020-09-08T17:06:25.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:06:25.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:06:25.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:06:25.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:06:25.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:06:25.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:06:25.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:06:25.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:06:25.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:06:25.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:06:25.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:06:25.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:06:25.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:06:25.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:06:25.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:06:25.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:06:25.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:06:25.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:06:25.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:06:25.759 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:06:25.760 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:06:25.760 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:06:25.760 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:06:25.760 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:25.761 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:25.761 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:25.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:25.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:06:25.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:06:25.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:06:25.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:06:25.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:06:25.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:06:25.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:06:25.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:25.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:06:25.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:06:25.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:06:25.763 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:06:25.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=71530084-7918-4ff2-b657-4c0fab058757 2020-09-08T17:06:25.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 71530084-7918-4ff2-b657-4c0fab058757' 2020-09-08T17:06:25.764 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:06:25.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQARuldfXsUVLhAABk/Vg+9zzj5HJ5ABZG3lRQ== 2020-09-08T17:06:25.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQARuldfXsUVLhAABk/Vg+9zzj5HJ5ABZG3lRQ=="}' 2020-09-08T17:06:25.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 71530084-7918-4ff2-b657-4c0fab058757 -i td/test-erasure-eio/2/new.json 2020-09-08T17:06:26.381 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:06:26.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:06:26.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQARuldfXsUVLhAABk/Vg+9zzj5HJ5ABZG3lRQ== --osd-uuid 71530084-7918-4ff2-b657-4c0fab058757 2020-09-08T17:06:26.413 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:26.418+0000 7f886b6beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:26.414 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:26.420+0000 7f886b6beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:26.416 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:26.421+0000 7f886b6beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:26.416 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:26.422+0000 7f886b6beec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:06:26.417 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:26.422+0000 7f886b6beec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:06:26.417 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:26.422+0000 7f886b6beec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:06:26.454 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:26.459+0000 7f886b6beec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:06:30.453 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:06:30.454 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:06:30.455 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:06:30.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:06:30.455 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:06:30.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:06:30.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:30.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:06:30.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:06:30.951 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:06:30.951 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:06:30.968 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:30.974+0000 7f9e8542eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:30.969 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:30.975+0000 7f9e8542eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:30.971 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:30.977+0000 7f9e8542eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:31.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:06:31.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:06:31.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:06:31.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:06:31.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:06:31.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:31.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:06:31.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:31.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:06:31.350 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:06:31.490 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:31.496+0000 7f9e8542eec0 -1 Falling back to public interface 2020-09-08T17:06:31.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:06:32.314 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:32.319+0000 7f9e8542eec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:06:32.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:06:32.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:32.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:06:32.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:32.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:06:32.721 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:06:33.115 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/141910,v1:127.0.0.1:6819/141910] [v2:127.0.0.1:6820/141910,v1:127.0.0.1:6821/141910] exists,up 71530084-7918-4ff2-b657-4c0fab058757 2020-09-08T17:06:33.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:06:33.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:06:33.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:06:33.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:06:33.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:06:33.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:06:33.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:06:33.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:06:33.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:06:33.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:06:33.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:06:33.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:06:33.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:06:33.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:06:33.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:06:33.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:06:33.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:06:33.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:06:33.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:06:33.119 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:06:33.119 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:06:33.120 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:06:33.120 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:06:33.120 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:33.120 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:33.120 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:33.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:33.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:06:33.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:06:33.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:06:33.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:06:33.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:06:33.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:06:33.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:06:33.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:33.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:06:33.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:06:33.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:06:33.123 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:06:33.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:06:33.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 0b20126f-2127-4301-9313-e8f94c5d453a' 2020-09-08T17:06:33.124 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:06:33.124 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:06:33.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAZuldfI2wGCBAA1tFWWZn27lnsdWNr14HJlg== 2020-09-08T17:06:33.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAZuldfI2wGCBAA1tFWWZn27lnsdWNr14HJlg=="}' 2020-09-08T17:06:33.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0b20126f-2127-4301-9313-e8f94c5d453a -i td/test-erasure-eio/3/new.json 2020-09-08T17:06:33.907 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:06:33.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:06:33.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAZuldfI2wGCBAA1tFWWZn27lnsdWNr14HJlg== --osd-uuid 0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:06:33.939 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:33.945+0000 7f8d17270ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:33.982 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:33.988+0000 7f8d17270ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:33.984 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:33.990+0000 7f8d17270ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:33.986 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:33.991+0000 7f8d17270ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:06:33.986 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:33.991+0000 7f8d17270ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:06:33.986 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:33.991+0000 7f8d17270ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:06:34.086 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:34.092+0000 7f8d17270ec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:06:38.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:06:38.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:06:38.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:06:38.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:06:38.781 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:06:39.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:06:39.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:39.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:06:39.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:06:39.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:06:39.279 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:06:39.297 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:39.303+0000 7f735a619ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:39.327 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:39.333+0000 7f735a619ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:39.328 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:39.334+0000 7f735a619ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:39.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:06:39.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:06:39.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:06:39.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:06:39.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:06:39.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:39.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:06:39.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:39.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:06:39.643 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:06:39.846 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:39.851+0000 7f735a619ec0 -1 Falling back to public interface 2020-09-08T17:06:40.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:06:40.662 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:40.667+0000 7f735a619ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:06:41.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:06:41.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:41.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:06:41.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:41.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:06:41.007 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:06:41.445 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:06:42.447 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:06:42.447 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:42.447 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:06:42.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:42.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:06:42.448 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:06:42.911 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/142912,v1:127.0.0.1:6827/142912] [v2:127.0.0.1:6828/142912,v1:127.0.0.1:6829/142912] exists,up 0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:06:42.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:06:42.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:06:42.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:06:42.912 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:06:42.913 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:06:42.913 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:06:42.913 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:06:42.913 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:42.913 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:42.914 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:06:42.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:06:42.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:06:43.001 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:06:43.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:06:43.011 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:06:16.910+0000 7ff2a93e1ec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:06:43.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:304: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2020-09-08T17:06:43.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2020-09-08T17:06:43.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:06:43.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:06:43.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=2 2020-09-08T17:06:43.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:06:43.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=1 2020-09-08T17:06:43.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:06:43.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2020-09-08T17:06:43.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:06:43.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:06:44.394 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:06:44.401 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:06:45.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:06:45.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:06:45.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:06:45.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:06:45.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:06:45.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:06:45.405 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:06:45.405 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:06:45.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:06:45.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:06:45.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:06:45.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:06:45.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:06:45.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:06:45.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:06:45.510 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:06:45.872 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:06:45.872 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:06:45.872 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:06:45.873 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:06:45.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:06:45.873 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:45.873 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:06:45.995 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803785 2020-09-08T17:06:45.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803785 2020-09-08T17:06:45.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785' 2020-09-08T17:06:45.996 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:45.996 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:06:46.113 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607558 2020-09-08T17:06:46.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607558 2020-09-08T17:06:46.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607558' 2020-09-08T17:06:46.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:46.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:06:46.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411334 2020-09-08T17:06:46.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411334 2020-09-08T17:06:46.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607558 2-77309411334' 2020-09-08T17:06:46.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:46.236 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:06:46.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149699 2020-09-08T17:06:46.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149699 2020-09-08T17:06:46.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607558 2-77309411334 3-111669149699' 2020-09-08T17:06:46.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:46.357 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803785 2020-09-08T17:06:46.357 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:46.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:06:46.357 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803785 2020-09-08T17:06:46.358 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:46.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803785 2020-09-08T17:06:46.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2020-09-08T17:06:46.359 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:06:46.359 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803785 2020-09-08T17:06:46.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803783 -lt 25769803785 2020-09-08T17:06:46.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:06:47.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:06:47.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:06:48.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803785 -lt 25769803785 2020-09-08T17:06:48.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:48.093 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607558 2020-09-08T17:06:48.093 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:48.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:06:48.094 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607558 2020-09-08T17:06:48.094 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:48.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607558 2020-09-08T17:06:48.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607558' 2020-09-08T17:06:48.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:06:48.095 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 51539607558 2020-09-08T17:06:48.456 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607559 -lt 51539607558 2020-09-08T17:06:48.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:48.457 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411334 2020-09-08T17:06:48.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:48.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:06:48.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411334 2020-09-08T17:06:48.459 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:48.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411334 2020-09-08T17:06:48.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411334' 2020-09-08T17:06:48.460 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:06:48.460 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411334 2020-09-08T17:06:48.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411334 -lt 77309411334 2020-09-08T17:06:48.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:48.824 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149699 2020-09-08T17:06:48.825 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:48.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:06:48.825 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149699 2020-09-08T17:06:48.826 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:48.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149699 2020-09-08T17:06:48.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2020-09-08T17:06:48.827 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:06:48.827 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149699 2020-09-08T17:06:49.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149700 -lt 111669149699 2020-09-08T17:06:49.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:06:49.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:06:49.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:06:49.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:06:49.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:06:49.675 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:06:49.675 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:06:49.676 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:06:49.676 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:06:49.676 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:06:49.676 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:06:50.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:06:50.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:06:50.026 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:06:50.026 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:06:50.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:06:50.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:06:50.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:06:50.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:307: TEST_rados_get_bad_size_shard_0: local shard_id=0 2020-09-08T17:06:50.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2020-09-08T17:06:50.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:221: rados_get_data_bad_size: local dir=td/test-erasure-eio 2020-09-08T17:06:50.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: shift 2020-09-08T17:06:50.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: local shard_id=0 2020-09-08T17:06:50.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: shift 2020-09-08T17:06:50.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: local bytes=10 2020-09-08T17:06:50.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: shift 2020-09-08T17:06:50.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: local mode=set 2020-09-08T17:06:50.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:229: rados_get_data_bad_size: local poolname=pool-jerasure 2020-09-08T17:06:50.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local objname=obj-size-59146-0-10 2020-09-08T17:06:50.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-59146-0-10 2020-09-08T17:06:50.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:06:50.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:06:50.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-size-59146-0-10 2020-09-08T17:06:50.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:06:50.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:06:50.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:06:50.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:06:50.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:06:50.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:06:50.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:06:50.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:06:50.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-size-59146-0-10 td/test-erasure-eio/ORIGINAL 2020-09-08T17:06:50.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: set_size obj-size-59146-0-10 td/test-erasure-eio 0 10 set 2020-09-08T17:06:50.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-0-10 2020-09-08T17:06:50.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:06:50.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:06:50.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:06:50.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=0 2020-09-08T17:06:50.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:06:50.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=10 2020-09-08T17:06:50.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:06:50.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:06:50.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:06:50.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:06:50.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-0-10 2020-09-08T17:06:50.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:06:50.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-0-10 2020-09-08T17:06:50.655 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-0-10 2020-09-08T17:06:50.655 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:06:51.018 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:06:51.019 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:06:51.019 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:06:51.019 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:06:51.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:06:51.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=3 2020-09-08T17:06:51.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:06:51.753 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:06:51.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:06:51.761 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 10 = 0 ']' 2020-09-08T17:06:51.762 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2020-09-08T17:06:51.763 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:06:51.764 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:06:51.764 INFO:tasks.workunit.client.0.smithi120.stderr:10 bytes copied, 6.6435e-05 s, 151 kB/s 2020-09-08T17:06:51.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:06:51.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:06:51.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:06:51.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=3 2020-09-08T17:06:51.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:06:51.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:06:51.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:06:51.765 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:06:51.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=3 2020-09-08T17:06:51.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:06:51.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2020-09-08T17:06:51.766 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:06:51.766 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:06:51.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:06:51.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:06:51.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:06:51.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:06:51.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:06:51.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:06:51.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:06:51.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=3 2020-09-08T17:06:51.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:06:51.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2020-09-08T17:06:51.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:06:53.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2020-09-08T17:06:53.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:06:53.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:06:53.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-09-08T17:06:53.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:06:53.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:06:53.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:06:53.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:06:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:06:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:06:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:06:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:06:53.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:06:53.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:06:53.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:06:53.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:06:53.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:06:53.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:06:53.926 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:06:53.926 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:06:53.926 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:06:53.926 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:53.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:06:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:06:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:06:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:06:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:06:53.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:06:53.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:53.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:06:53.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:06:53.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:06:53.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-09-08T17:06:53.930 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.3 2020-09-08T17:06:53.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:06:53.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/3/whoami 2020-09-08T17:06:53.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-09-08T17:06:53.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:06:53.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:06:53.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:06:53.944 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:53.951+0000 7f3265c8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:53.946 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:53.953+0000 7f3265c8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:53.949 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:53.955+0000 7f3265c8dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:06:54.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-09-08T17:06:54.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:06:54.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:06:54.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:06:54.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:06:54.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:54.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:06:54.309 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:06:54.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:54.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:06:54.465 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:54.471+0000 7f3265c8dec0 -1 Falling back to public interface 2020-09-08T17:06:54.674 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:06:55.353 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:06:55.360+0000 7f3265c8dec0 -1 osd.3 34 log_to_monitors {default=true} 2020-09-08T17:06:55.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:06:55.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:06:55.675 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:06:55.676 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:06:55.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:06:55.676 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:06:56.087 INFO:tasks.workunit.client.0.smithi120.stderr:osd.3 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [26,34) [v2:127.0.0.1:6826/144188,v1:127.0.0.1:6827/144188] [v2:127.0.0.1:6828/144188,v1:127.0.0.1:6829/144188] exists,up 0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:06:56.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:06:56.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:06:56.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:06:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:06:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:06:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:06:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:06:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:06:56.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:06:56.090 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:06:56.090 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:06:56.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:06:56.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:06:56.091 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:06:56.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:06:56.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:06:56.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:06:56.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:06:56.181 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:06:56.684 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:06:56.684 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:06:56.684 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:06:56.684 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:06:56.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:06:56.685 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:56.685 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:06:56.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803789 2020-09-08T17:06:56.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803789 2020-09-08T17:06:56.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789' 2020-09-08T17:06:56.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:56.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:06:56.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607563 2020-09-08T17:06:56.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607563 2020-09-08T17:06:56.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607563' 2020-09-08T17:06:56.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:56.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:06:57.050 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411338 2020-09-08T17:06:57.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411338 2020-09-08T17:06:57.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607563 2-77309411338' 2020-09-08T17:06:57.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:06:57.051 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:06:57.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724547 2020-09-08T17:06:57.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724547 2020-09-08T17:06:57.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803789 1-51539607563 2-77309411338 3-167503724547' 2020-09-08T17:06:57.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:57.172 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803789 2020-09-08T17:06:57.172 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:57.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:06:57.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803789 2020-09-08T17:06:57.173 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:57.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803789 2020-09-08T17:06:57.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2020-09-08T17:06:57.174 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803789 2020-09-08T17:06:57.174 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:06:57.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803787 -lt 25769803789 2020-09-08T17:06:57.535 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:06:58.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:06:58.537 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:06:58.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803789 -lt 25769803789 2020-09-08T17:06:58.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:58.910 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607563 2020-09-08T17:06:58.910 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:58.911 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:06:58.911 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607563 2020-09-08T17:06:58.911 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:58.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607563 2020-09-08T17:06:58.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607563' 2020-09-08T17:06:58.912 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 51539607563 2020-09-08T17:06:58.912 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:06:59.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607563 -lt 51539607563 2020-09-08T17:06:59.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:59.279 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411338 2020-09-08T17:06:59.279 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:59.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:06:59.280 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411338 2020-09-08T17:06:59.280 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:59.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411338 2020-09-08T17:06:59.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411338' 2020-09-08T17:06:59.282 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411338 2020-09-08T17:06:59.282 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:06:59.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411338 -lt 77309411338 2020-09-08T17:06:59.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:06:59.644 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-167503724547 2020-09-08T17:06:59.645 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:06:59.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:06:59.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-167503724547 2020-09-08T17:06:59.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:06:59.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724547 2020-09-08T17:06:59.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 167503724547' 2020-09-08T17:06:59.647 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 167503724547 2020-09-08T17:06:59.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:07:00.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724547 -lt 167503724547 2020-09-08T17:07:00.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:07:00.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:00.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:00.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:07:00.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:07:00.492 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:07:00.492 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:07:00.493 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:07:00.493 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:07:00.493 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:07:00.493 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:07:00.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:07:00.843 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:07:00.843 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:00.843 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:01.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:07:01.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:07:01.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:07:01.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:07:01.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:07:02.098 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:07:02.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-0-10 2020-09-08T17:07:02.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:07:02.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:07:02.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-0-10 2020-09-08T17:07:02.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:07:02.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:07:02.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-size-59146-0-10 td/test-erasure-eio/COPY 2020-09-08T17:07:08.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:07:08.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:07:08.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: expr 0 + 1 2020-09-08T17:07:08.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: shard_id=1 2020-09-08T17:07:08.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: set_size obj-size-59146-0-10 td/test-erasure-eio 1 10 set 2020-09-08T17:07:08.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-0-10 2020-09-08T17:07:08.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:07:08.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:07:08.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:07:08.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=1 2020-09-08T17:07:08.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:07:08.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=10 2020-09-08T17:07:08.966 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:07:08.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:07:08.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:07:08.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:07:08.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-0-10 2020-09-08T17:07:08.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:07:08.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-0-10 2020-09-08T17:07:08.968 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-0-10 2020-09-08T17:07:08.968 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:07:09.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:07:09.369 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:09.369 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:07:09.369 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:07:09.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:07:09.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=1 2020-09-08T17:07:09.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:07:10.203 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:07:10.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:07:10.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 10 = 0 ']' 2020-09-08T17:07:10.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2020-09-08T17:07:10.213 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:07:10.214 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:07:10.214 INFO:tasks.workunit.client.0.smithi120.stderr:10 bytes copied, 4.3121e-05 s, 232 kB/s 2020-09-08T17:07:10.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:10.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:07:10.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:07:10.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:07:10.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:07:10.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:10.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:07:10.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:07:10.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:07:10.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:07:10.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:07:10.216 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:07:10.216 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:07:10.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:07:10.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:07:10.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:07:10.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:07:10.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:10.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:07:10.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:07:10.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:07:10.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:07:10.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:07:10.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-0-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:12.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:07:12.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:07:12.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:07:12.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:07:12.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:07:12.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:07:12.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:07:12.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:07:12.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:07:12.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:07:12.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:07:12.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:07:12.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:07:12.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:07:12.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:07:12.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:07:12.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:07:12.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:07:12.433 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:07:12.433 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:07:12.434 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:07:12.434 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:12.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:12.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:07:12.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:07:12.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:07:12.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:07:12.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:07:12.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:12.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:07:12.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:07:12.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:07:12.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:07:12.437 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:07:12.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:12.438 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:07:12.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:07:12.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:07:12.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:07:12.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:07:12.451 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:12.457+0000 7f579b873ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:12.453 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:12.459+0000 7f579b873ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:12.454 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:12.460+0000 7f579b873ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:12.805 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:07:12.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:07:12.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:07:12.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:07:12.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:07:12.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:12.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:07:12.807 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:07:12.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:12.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:07:12.974 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:12.980+0000 7f579b873ec0 -1 Falling back to public interface 2020-09-08T17:07:13.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:07:13.916 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:13.922+0000 7f579b873ec0 -1 osd.1 43 log_to_monitors {default=true} 2020-09-08T17:07:14.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:07:14.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:14.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:07:14.210 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:14.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:14.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:07:14.595 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 49 up_thru 36 down_at 46 last_clean_interval [12,43) [v2:127.0.0.1:6810/145410,v1:127.0.0.1:6811/145410] [v2:127.0.0.1:6812/145410,v1:127.0.0.1:6813/145410] exists,up 285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:07:14.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:07:14.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:07:14.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:07:14.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:07:14.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:07:14.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:07:14.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:07:14.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:07:14.597 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:07:14.598 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:07:14.598 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:07:14.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:07:14.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:07:14.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:07:14.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:07:14.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:07:14.687 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:07:14.688 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:07:14.688 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:07:15.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:07:15.116 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:15.116 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:07:15.117 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:07:15.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:07:15.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:15.117 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:07:15.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803795 2020-09-08T17:07:15.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803795 2020-09-08T17:07:15.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795' 2020-09-08T17:07:15.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:15.237 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:07:15.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397507 2020-09-08T17:07:15.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397507 2020-09-08T17:07:15.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-210453397507' 2020-09-08T17:07:15.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:15.358 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:07:15.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411344 2020-09-08T17:07:15.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411344 2020-09-08T17:07:15.478 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-210453397507 2-77309411344' 2020-09-08T17:07:15.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:15.479 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:07:15.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724553 2020-09-08T17:07:15.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724553 2020-09-08T17:07:15.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-210453397507 2-77309411344 3-167503724553' 2020-09-08T17:07:15.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:15.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803795 2020-09-08T17:07:15.600 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:15.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:07:15.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803795 2020-09-08T17:07:15.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:15.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803795 2020-09-08T17:07:15.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803795' 2020-09-08T17:07:15.602 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803795 2020-09-08T17:07:15.602 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:07:15.961 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803795 -lt 25769803795 2020-09-08T17:07:15.962 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:15.962 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397507 2020-09-08T17:07:15.963 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:15.963 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:07:15.963 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397507 2020-09-08T17:07:15.964 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:15.964 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397507 2020-09-08T17:07:15.965 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397507' 2020-09-08T17:07:15.965 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 210453397507 2020-09-08T17:07:15.965 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:07:16.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397507 -lt 210453397507 2020-09-08T17:07:16.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:16.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411344 2020-09-08T17:07:16.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:16.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:07:16.354 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411344 2020-09-08T17:07:16.354 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:16.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411344 2020-09-08T17:07:16.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411344' 2020-09-08T17:07:16.356 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411344 2020-09-08T17:07:16.356 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:07:16.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411344 -lt 77309411344 2020-09-08T17:07:16.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:16.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-167503724553 2020-09-08T17:07:16.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:16.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:07:16.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-167503724553 2020-09-08T17:07:16.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:16.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724553 2020-09-08T17:07:16.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 167503724553' 2020-09-08T17:07:16.723 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 167503724553 2020-09-08T17:07:16.723 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:07:17.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724551 -lt 167503724553 2020-09-08T17:07:17.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:07:18.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:07:18.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:07:18.485 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724553 -lt 167503724553 2020-09-08T17:07:18.486 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:07:18.486 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:18.487 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:18.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:07:18.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:07:18.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:07:18.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:07:18.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:07:18.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:07:18.978 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:07:18.978 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:07:19.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T17:07:19.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:07:19.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:19.327 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:19.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T17:07:19.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' -1 2020-09-08T17:07:19.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T17:07:19.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=4 2020-09-08T17:07:19.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:07:19.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T17:07:19.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:07:19.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:07:19.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:07:19.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:07:19.923 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:07:19.923 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:07:19.923 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:07:19.923 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:07:20.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:07:20.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:07:20.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:20.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:20.758 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:07:20.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:07:20.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:07:20.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:07:20.760 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:07:21.449 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:07:21.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-0-10 fail 2020-09-08T17:07:21.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:07:21.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:07:21.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-0-10 2020-09-08T17:07:21.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:07:21.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:07:21.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-size-59146-0-10 td/test-erasure-eio/COPY 2020-09-08T17:07:21.576 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-size-59146-0-10: (5) Input/output error 2020-09-08T17:07:21.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:07:21.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:07:21.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2020-09-08T17:07:21.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:221: rados_get_data_bad_size: local dir=td/test-erasure-eio 2020-09-08T17:07:21.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: shift 2020-09-08T17:07:21.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: local shard_id=0 2020-09-08T17:07:21.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: shift 2020-09-08T17:07:21.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: local bytes=0 2020-09-08T17:07:21.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: shift 2020-09-08T17:07:21.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: local mode=set 2020-09-08T17:07:21.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:229: rados_get_data_bad_size: local poolname=pool-jerasure 2020-09-08T17:07:21.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local objname=obj-size-59146-0-0 2020-09-08T17:07:21.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-59146-0-0 2020-09-08T17:07:21.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:07:21.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:07:21.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-size-59146-0-0 2020-09-08T17:07:21.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:21.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:07:21.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:21.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:07:21.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:21.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:07:21.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:21.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:07:21.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-size-59146-0-0 td/test-erasure-eio/ORIGINAL 2020-09-08T17:07:21.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: set_size obj-size-59146-0-0 td/test-erasure-eio 0 0 set 2020-09-08T17:07:21.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-0-0 2020-09-08T17:07:21.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:07:21.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:07:21.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:07:21.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=0 2020-09-08T17:07:21.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:07:21.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=0 2020-09-08T17:07:21.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:07:21.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:07:21.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:07:21.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:07:21.720 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-0-0 2020-09-08T17:07:21.720 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:07:21.720 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-0-0 2020-09-08T17:07:21.720 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-0-0 2020-09-08T17:07:21.721 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:07:22.109 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:07:22.110 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:22.110 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:07:22.110 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:07:22.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:07:22.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=3 2020-09-08T17:07:22.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:07:22.804 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:07:22.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:07:22.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 0 = 0 ']' 2020-09-08T17:07:22.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:211: set_size: touch td/test-erasure-eio/CORRUPT 2020-09-08T17:07:22.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:22.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:07:22.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:07:22.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=3 2020-09-08T17:07:22.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:07:22.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:22.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:07:22.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:07:22.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=3 2020-09-08T17:07:22.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:07:22.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2020-09-08T17:07:22.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:07:22.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:07:22.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:07:22.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:07:22.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:07:22.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:07:22.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:22.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:07:22.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:07:22.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=3 2020-09-08T17:07:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:07:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2020-09-08T17:07:22.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:25.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2020-09-08T17:07:25.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:07:25.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:07:25.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-09-08T17:07:25.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:07:25.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:07:25.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:07:25.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:07:25.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:07:25.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:07:25.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:07:25.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:07:25.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:07:25.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:07:25.015 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:07:25.016 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:07:25.016 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:07:25.016 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:07:25.016 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:07:25.016 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:07:25.016 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:07:25.017 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:25.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:25.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:07:25.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:07:25.017 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:07:25.018 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:07:25.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:07:25.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:25.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:07:25.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:07:25.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:07:25.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-09-08T17:07:25.025 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.3 2020-09-08T17:07:25.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:25.026 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/3/whoami 2020-09-08T17:07:25.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-09-08T17:07:25.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:07:25.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:07:25.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:07:25.034 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:25.040+0000 7fd2408a8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:25.036 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:25.042+0000 7fd2408a8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:25.038 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:25.044+0000 7fd2408a8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:25.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-09-08T17:07:25.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:07:25.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:07:25.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:07:25.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:07:25.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:25.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:07:25.395 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:07:25.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:25.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:07:25.556 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:25.561+0000 7fd2408a8ec0 -1 Falling back to public interface 2020-09-08T17:07:25.754 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:07:26.493 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:26.499+0000 7fd2408a8ec0 -1 osd.3 54 log_to_monitors {default=true} 2020-09-08T17:07:26.755 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:07:26.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:26.756 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:07:26.756 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:26.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:26.757 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:07:27.210 INFO:tasks.workunit.client.0.smithi120.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [39,54) [v2:127.0.0.1:6826/146703,v1:127.0.0.1:6827/146703] [v2:127.0.0.1:6828/146703,v1:127.0.0.1:6829/146703] exists,up 0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:07:27.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:07:27.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:07:27.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:07:27.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:07:27.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:07:27.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:07:27.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:07:27.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:07:27.212 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:07:27.212 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:07:27.212 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:07:27.213 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:07:27.213 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:07:27.213 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:07:27.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:07:27.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:07:27.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:07:27.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:07:27.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:07:27.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:07:27.665 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:27.665 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:07:27.666 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:07:27.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:07:27.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:27.666 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:07:27.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803799 2020-09-08T17:07:27.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803799 2020-09-08T17:07:27.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799' 2020-09-08T17:07:27.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:27.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:07:27.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397511 2020-09-08T17:07:27.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397511 2020-09-08T17:07:27.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-210453397511' 2020-09-08T17:07:27.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:27.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:07:28.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411348 2020-09-08T17:07:28.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411348 2020-09-08T17:07:28.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-210453397511 2-77309411348' 2020-09-08T17:07:28.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:28.066 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:07:28.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=253403070467 2020-09-08T17:07:28.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 253403070467 2020-09-08T17:07:28.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803799 1-210453397511 2-77309411348 3-253403070467' 2020-09-08T17:07:28.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:28.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803799 2020-09-08T17:07:28.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:28.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:07:28.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803799 2020-09-08T17:07:28.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:28.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803799 2020-09-08T17:07:28.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803799' 2020-09-08T17:07:28.190 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803799 2020-09-08T17:07:28.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:07:28.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803797 -lt 25769803799 2020-09-08T17:07:28.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:07:29.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:07:29.553 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:07:29.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803800 -lt 25769803799 2020-09-08T17:07:29.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:29.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397511 2020-09-08T17:07:29.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:29.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:07:29.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397511 2020-09-08T17:07:29.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:29.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397511 2020-09-08T17:07:29.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397511' 2020-09-08T17:07:29.931 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 210453397511 2020-09-08T17:07:29.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:07:30.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397512 -lt 210453397511 2020-09-08T17:07:30.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:30.292 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411348 2020-09-08T17:07:30.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:30.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:07:30.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411348 2020-09-08T17:07:30.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:30.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411348 2020-09-08T17:07:30.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411348' 2020-09-08T17:07:30.295 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411348 2020-09-08T17:07:30.295 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:07:30.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411349 -lt 77309411348 2020-09-08T17:07:30.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:30.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-253403070467 2020-09-08T17:07:30.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:30.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:07:30.658 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-253403070467 2020-09-08T17:07:30.659 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:30.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=253403070467 2020-09-08T17:07:30.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 253403070467' 2020-09-08T17:07:30.660 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 253403070467 2020-09-08T17:07:30.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:07:31.019 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 253403070467 -lt 253403070467 2020-09-08T17:07:31.019 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:07:31.020 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:31.020 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:31.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:07:31.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:07:31.510 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:07:31.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:07:31.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:07:31.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:07:31.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:07:31.511 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:07:31.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:07:31.862 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:07:31.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:31.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:32.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:07:32.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:07:32.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:07:32.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:07:32.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:07:33.012 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:07:33.020 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-0-0 2020-09-08T17:07:33.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:07:33.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:07:33.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-0-0 2020-09-08T17:07:33.021 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:07:33.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:07:33.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-size-59146-0-0 td/test-erasure-eio/COPY 2020-09-08T17:07:39.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:07:39.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:07:39.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: expr 0 + 1 2020-09-08T17:07:39.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: shard_id=1 2020-09-08T17:07:39.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: set_size obj-size-59146-0-0 td/test-erasure-eio 1 0 set 2020-09-08T17:07:39.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-0-0 2020-09-08T17:07:39.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:07:39.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:07:39.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:07:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=1 2020-09-08T17:07:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:07:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=0 2020-09-08T17:07:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:07:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:07:39.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:07:39.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:07:39.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-0-0 2020-09-08T17:07:39.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:07:39.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-0-0 2020-09-08T17:07:39.732 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-0-0 2020-09-08T17:07:39.732 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:07:40.107 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:07:40.107 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:40.108 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:07:40.108 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:07:40.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:07:40.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=1 2020-09-08T17:07:40.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:07:40.816 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:07:40.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:07:40.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 0 = 0 ']' 2020-09-08T17:07:40.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:211: set_size: touch td/test-erasure-eio/CORRUPT 2020-09-08T17:07:40.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:40.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:07:40.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:07:40.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:07:40.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:07:40.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:40.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:07:40.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:07:40.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:07:40.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:07:40.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:07:40.829 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:07:40.829 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:07:40.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:07:40.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:07:40.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:07:40.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:07:40.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:40.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:07:40.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:07:40.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:07:40.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:07:40.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:07:40.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-0-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:42.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:07:42.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:07:42.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:07:42.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:07:42.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:07:42.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:07:42.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:07:42.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:07:42.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:07:42.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:07:42.919 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:07:42.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:07:42.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:07:42.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:07:42.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:07:42.920 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:07:42.921 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:07:42.921 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:07:42.921 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:07:42.921 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:07:42.921 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:07:42.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:42.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:42.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:07:42.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:07:42.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:07:42.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:07:42.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:07:42.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:42.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:07:42.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:07:42.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:07:42.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:07:42.925 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:07:42.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:42.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:07:42.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:07:42.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:07:42.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:07:42.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:07:42.939 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:42.946+0000 7fa69048cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:42.941 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:42.948+0000 7fa69048cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:42.944 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:42.951+0000 7fa69048cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:43.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:07:43.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:07:43.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:07:43.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:07:43.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:07:43.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:43.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:07:43.291 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:07:43.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:43.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:07:43.460 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:43.466+0000 7fa69048cec0 -1 Falling back to public interface 2020-09-08T17:07:43.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:07:44.377 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:44.384+0000 7fa69048cec0 -1 osd.1 62 log_to_monitors {default=true} 2020-09-08T17:07:44.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:07:44.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:44.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:07:44.654 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:44.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:07:44.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:45.052 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [49,62) [v2:127.0.0.1:6810/147933,v1:127.0.0.1:6811/147933] [v2:127.0.0.1:6812/147933,v1:127.0.0.1:6813/147933] exists,up 285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:07:45.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:07:45.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:07:45.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:07:45.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:07:45.053 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:07:45.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:07:45.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:07:45.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:07:45.054 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:07:45.054 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:07:45.055 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:07:45.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:07:45.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:07:45.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:07:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:07:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:07:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:07:45.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:07:45.145 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:07:45.677 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:07:45.677 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:45.678 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:07:45.678 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:07:45.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:07:45.678 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:45.678 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:07:45.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803805 2020-09-08T17:07:45.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803805 2020-09-08T17:07:45.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805' 2020-09-08T17:07:45.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:45.825 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:07:45.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=292057776131 2020-09-08T17:07:45.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 292057776131 2020-09-08T17:07:45.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-292057776131' 2020-09-08T17:07:45.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:45.945 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:07:46.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411354 2020-09-08T17:07:46.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411354 2020-09-08T17:07:46.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-292057776131 2-77309411354' 2020-09-08T17:07:46.064 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:46.065 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:07:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=253403070472 2020-09-08T17:07:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 253403070472 2020-09-08T17:07:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-292057776131 2-77309411354 3-253403070472' 2020-09-08T17:07:46.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:46.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803805 2020-09-08T17:07:46.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:46.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:07:46.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803805 2020-09-08T17:07:46.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:46.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803805 2020-09-08T17:07:46.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2020-09-08T17:07:46.188 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803805 2020-09-08T17:07:46.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:07:46.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803803 -lt 25769803805 2020-09-08T17:07:46.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:07:47.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:07:47.555 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:07:47.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803805 -lt 25769803805 2020-09-08T17:07:47.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:47.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-292057776131 2020-09-08T17:07:47.926 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:47.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:07:47.927 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-292057776131 2020-09-08T17:07:47.927 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:47.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=292057776131 2020-09-08T17:07:47.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2020-09-08T17:07:47.928 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 292057776131 2020-09-08T17:07:47.928 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:07:48.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 292057776131 -lt 292057776131 2020-09-08T17:07:48.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:48.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411354 2020-09-08T17:07:48.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:48.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:07:48.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411354 2020-09-08T17:07:48.295 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:48.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411354 2020-09-08T17:07:48.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411354' 2020-09-08T17:07:48.296 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411354 2020-09-08T17:07:48.296 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:07:48.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411354 -lt 77309411354 2020-09-08T17:07:48.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:48.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-253403070472 2020-09-08T17:07:48.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:48.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:07:48.661 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-253403070472 2020-09-08T17:07:48.661 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:48.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=253403070472 2020-09-08T17:07:48.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 253403070472' 2020-09-08T17:07:48.662 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 253403070472 2020-09-08T17:07:48.663 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:07:49.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 253403070473 -lt 253403070472 2020-09-08T17:07:49.031 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:07:49.031 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:49.031 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:49.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:07:49.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:07:49.513 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:07:49.514 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:07:49.514 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:07:49.514 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:07:49.514 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:07:49.514 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:07:49.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:07:49.869 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:07:49.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:49.870 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:50.386 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:07:50.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:07:50.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:07:50.387 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:07:50.388 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:07:51.042 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:07:51.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-0-0 fail 2020-09-08T17:07:51.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:07:51.051 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:07:51.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-0-0 2020-09-08T17:07:51.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:07:51.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:07:51.052 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-size-59146-0-0 td/test-erasure-eio/COPY 2020-09-08T17:07:51.166 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-size-59146-0-0: (5) Input/output error 2020-09-08T17:07:51.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:07:51.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:07:51.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2020-09-08T17:07:51.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:221: rados_get_data_bad_size: local dir=td/test-erasure-eio 2020-09-08T17:07:51.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: shift 2020-09-08T17:07:51.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: local shard_id=0 2020-09-08T17:07:51.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: shift 2020-09-08T17:07:51.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: local bytes=256 2020-09-08T17:07:51.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: shift 2020-09-08T17:07:51.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: local mode=add 2020-09-08T17:07:51.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:229: rados_get_data_bad_size: local poolname=pool-jerasure 2020-09-08T17:07:51.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local objname=obj-size-59146-0-256 2020-09-08T17:07:51.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-59146-0-256 2020-09-08T17:07:51.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:07:51.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:07:51.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-size-59146-0-256 2020-09-08T17:07:51.172 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:51.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:07:51.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:51.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:07:51.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:51.173 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:07:51.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:07:51.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:07:51.174 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-size-59146-0-256 td/test-erasure-eio/ORIGINAL 2020-09-08T17:07:51.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: set_size obj-size-59146-0-256 td/test-erasure-eio 0 256 add 2020-09-08T17:07:51.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-0-256 2020-09-08T17:07:51.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:07:51.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:07:51.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:07:51.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=0 2020-09-08T17:07:51.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:07:51.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=256 2020-09-08T17:07:51.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:07:51.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=add 2020-09-08T17:07:51.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:07:51.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:07:51.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-0-256 2020-09-08T17:07:51.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:07:51.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-0-256 2020-09-08T17:07:51.289 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-0-256 2020-09-08T17:07:51.289 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:07:51.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:07:51.657 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:51.657 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:07:51.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:07:51.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:07:51.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=3 2020-09-08T17:07:51.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:07:52.322 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:07:52.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' add = add ']' 2020-09-08T17:07:52.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:52.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:07:52.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:07:52.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=3 2020-09-08T17:07:52.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:07:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:07:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:07:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=3 2020-09-08T17:07:52.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:07:52.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2020-09-08T17:07:52.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:07:52.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:07:52.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:07:52.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:07:52.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:07:52.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:07:52.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:52.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:07:52.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:07:52.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=3 2020-09-08T17:07:52.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:07:52.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2020-09-08T17:07:52.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:07:54.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2020-09-08T17:07:54.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:07:54.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:07:54.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-09-08T17:07:54.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:07:54.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:07:54.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:07:54.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:07:54.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:07:54.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:07:54.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:07:54.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:07:54.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:07:54.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:07:54.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:07:54.349 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:07:54.349 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:07:54.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:07:54.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:07:54.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:07:54.350 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:07:54.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:54.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:07:54.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:07:54.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:07:54.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:07:54.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:07:54.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:07:54.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:54.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:07:54.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:07:54.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:07:54.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-09-08T17:07:54.354 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.3 2020-09-08T17:07:54.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:07:54.354 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/3/whoami 2020-09-08T17:07:54.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-09-08T17:07:54.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:07:54.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:07:54.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:07:54.368 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:54.374+0000 7f4c19151ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:54.370 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:54.376+0000 7f4c19151ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:54.373 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:54.379+0000 7f4c19151ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:07:54.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-09-08T17:07:54.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:07:54.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:07:54.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:07:54.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:07:54.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:54.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:07:54.726 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:07:54.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:54.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:07:54.888 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:54.894+0000 7f4c19151ec0 -1 Falling back to public interface 2020-09-08T17:07:55.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:07:55.881 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:07:55.887+0000 7f4c19151ec0 -1 osd.3 72 log_to_monitors {default=true} 2020-09-08T17:07:56.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:07:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:07:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:07:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:56.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:07:56.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:07:56.501 INFO:tasks.workunit.client.0.smithi120.stderr:osd.3 up in weight 1 up_from 77 up_thru 77 down_at 75 last_clean_interval [59,72) [v2:127.0.0.1:6826/149172,v1:127.0.0.1:6827/149172] [v2:127.0.0.1:6828/149172,v1:127.0.0.1:6829/149172] exists,up 0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:07:56.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:07:56.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:07:56.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:07:56.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:07:56.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:07:56.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:07:56.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:07:56.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:07:56.503 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:07:56.504 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:07:56.504 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:07:56.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:07:56.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:07:56.504 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:07:56.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:07:56.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:07:56.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:07:56.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:07:56.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:07:57.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:07:57.030 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:07:57.030 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:07:57.031 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:07:57.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:07:57.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:57.031 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:07:57.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803809 2020-09-08T17:07:57.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803809 2020-09-08T17:07:57.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809' 2020-09-08T17:07:57.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:57.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:07:57.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=292057776135 2020-09-08T17:07:57.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 292057776135 2020-09-08T17:07:57.271 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809 1-292057776135' 2020-09-08T17:07:57.272 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:57.272 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:07:57.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411358 2020-09-08T17:07:57.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411358 2020-09-08T17:07:57.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809 1-292057776135 2-77309411358' 2020-09-08T17:07:57.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:07:57.391 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:07:57.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=330712481795 2020-09-08T17:07:57.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 330712481795 2020-09-08T17:07:57.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809 1-292057776135 2-77309411358 3-330712481795' 2020-09-08T17:07:57.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:57.512 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803809 2020-09-08T17:07:57.512 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:57.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:07:57.513 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803809 2020-09-08T17:07:57.513 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:57.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803809 2020-09-08T17:07:57.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2020-09-08T17:07:57.514 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803809 2020-09-08T17:07:57.514 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:07:57.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803809 -lt 25769803809 2020-09-08T17:07:57.906 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:57.907 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-292057776135 2020-09-08T17:07:57.907 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:57.908 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:07:57.908 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-292057776135 2020-09-08T17:07:57.909 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:57.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=292057776135 2020-09-08T17:07:57.909 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 292057776135' 2020-09-08T17:07:57.910 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 292057776135 2020-09-08T17:07:57.910 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:07:58.273 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 292057776135 -lt 292057776135 2020-09-08T17:07:58.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:58.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411358 2020-09-08T17:07:58.274 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:58.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:07:58.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411358 2020-09-08T17:07:58.275 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:58.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411358 2020-09-08T17:07:58.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411358' 2020-09-08T17:07:58.276 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411358 2020-09-08T17:07:58.277 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:07:58.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411358 -lt 77309411358 2020-09-08T17:07:58.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:07:58.643 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-330712481795 2020-09-08T17:07:58.643 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:07:58.644 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:07:58.644 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-330712481795 2020-09-08T17:07:58.644 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:07:58.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=330712481795 2020-09-08T17:07:58.645 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 330712481795' 2020-09-08T17:07:58.645 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 330712481795 2020-09-08T17:07:58.645 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:07:59.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 330712481795 -lt 330712481795 2020-09-08T17:07:59.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:07:59.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:59.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:07:59.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:07:59.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:07:59.501 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:07:59.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:07:59.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:07:59.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:07:59.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:07:59.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:07:59.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:07:59.860 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:07:59.860 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:07:59.860 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:08:00.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:08:00.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:08:00.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:08:00.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: dd if=/dev/urandom bs=256 count=1 2020-09-08T17:08:00.343 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:08:00.343 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:08:00.344 INFO:tasks.workunit.client.0.smithi120.stderr:256 bytes copied, 2.8863e-05 s, 8.9 MB/s 2020-09-08T17:08:00.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:00.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:08:00.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:08:00.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=3 2020-09-08T17:08:00.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:08:00.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:00.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:08:00.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:08:00.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=3 2020-09-08T17:08:00.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:08:00.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2020-09-08T17:08:00.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:08:00.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:08:00.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:08:00.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:08:00.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:08:00.447 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:08:00.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:00.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:08:00.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:08:00.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=3 2020-09-08T17:08:00.448 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:08:00.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2020-09-08T17:08:00.449 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:02.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2020-09-08T17:08:02.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:08:02.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:08:02.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2020-09-08T17:08:02.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:08:02.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:08:02.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:08:02.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:08:02.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:08:02.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:08:02.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:08:02.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:08:02.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:08:02.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:08:02.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:08:02.289 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:08:02.290 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:02.290 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:02.290 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:02.290 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:02.290 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:02.290 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:02.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:02.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:08:02.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:08:02.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:08:02.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:08:02.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:08:02.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:02.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:08:02.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:08:02.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:08:02.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2020-09-08T17:08:02.294 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.3 2020-09-08T17:08:02.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:02.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/3/whoami 2020-09-08T17:08:02.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2020-09-08T17:08:02.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:08:02.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:08:02.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:08:02.308 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:02.314+0000 7fc3b6846ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:02.310 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:02.316+0000 7fc3b6846ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:02.312 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:02.318+0000 7fc3b6846ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:02.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2020-09-08T17:08:02.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:08:02.663 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:08:02.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:08:02.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:08:02.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:02.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:08:02.665 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:08:02.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:02.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:08:02.828 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:02.833+0000 7fc3b6846ec0 -1 Falling back to public interface 2020-09-08T17:08:03.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:08:03.855 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:03.861+0000 7fc3b6846ec0 -1 osd.3 79 log_to_monitors {default=true} 2020-09-08T17:08:04.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:08:04.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:04.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:08:04.034 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:08:04.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:04.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:08:04.461 INFO:tasks.workunit.client.0.smithi120.stderr:osd.3 up in weight 1 up_from 82 up_thru 82 down_at 80 last_clean_interval [77,79) [v2:127.0.0.1:6826/150270,v1:127.0.0.1:6827/150270] [v2:127.0.0.1:6828/150270,v1:127.0.0.1:6829/150270] exists,up 0b20126f-2127-4301-9313-e8f94c5d453a 2020-09-08T17:08:04.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:08:04.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:08:04.461 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:08:04.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:08:04.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:08:04.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:08:04.462 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:08:04.463 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:08:04.463 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:08:04.463 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:08:04.463 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:08:04.463 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:08:04.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:08:04.464 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:08:04.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:08:04.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:08:04.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:08:04.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:08:04.552 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:08:04.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:08:04.937 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:08:04.937 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:08:04.938 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:08:04.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:08:04.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:04.938 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:08:05.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803813 2020-09-08T17:08:05.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803813 2020-09-08T17:08:05.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803813' 2020-09-08T17:08:05.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:05.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:08:05.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=292057776139 2020-09-08T17:08:05.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 292057776139 2020-09-08T17:08:05.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803813 1-292057776139' 2020-09-08T17:08:05.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:05.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:08:05.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411362 2020-09-08T17:08:05.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411362 2020-09-08T17:08:05.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803813 1-292057776139 2-77309411362' 2020-09-08T17:08:05.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:05.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:08:05.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318275 2020-09-08T17:08:05.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318275 2020-09-08T17:08:05.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803813 1-292057776139 2-77309411362 3-352187318275' 2020-09-08T17:08:05.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:05.419 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803813 2020-09-08T17:08:05.419 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:05.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:08:05.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803813 2020-09-08T17:08:05.421 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:05.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803813 2020-09-08T17:08:05.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2020-09-08T17:08:05.422 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803813 2020-09-08T17:08:05.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:08:05.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803813 -lt 25769803813 2020-09-08T17:08:05.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:05.787 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-292057776139 2020-09-08T17:08:05.787 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:05.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:08:05.788 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-292057776139 2020-09-08T17:08:05.788 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:05.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=292057776139 2020-09-08T17:08:05.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 292057776139' 2020-09-08T17:08:05.789 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 292057776139 2020-09-08T17:08:05.790 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:08:06.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 292057776139 -lt 292057776139 2020-09-08T17:08:06.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:06.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411362 2020-09-08T17:08:06.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:06.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:08:06.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411362 2020-09-08T17:08:06.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:06.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411362 2020-09-08T17:08:06.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411362' 2020-09-08T17:08:06.166 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411362 2020-09-08T17:08:06.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:08:06.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411362 -lt 77309411362 2020-09-08T17:08:06.536 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:06.537 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-352187318275 2020-09-08T17:08:06.537 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:06.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:08:06.538 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-352187318275 2020-09-08T17:08:06.538 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:06.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318275 2020-09-08T17:08:06.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 352187318275' 2020-09-08T17:08:06.539 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 352187318275 2020-09-08T17:08:06.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:08:06.904 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318275 -lt 352187318275 2020-09-08T17:08:06.904 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:08:06.905 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:08:06.905 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:08:07.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:08:07.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:08:07.395 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:08:07.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:08:07.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:08:07.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:08:07.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:08:07.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:08:07.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:08:07.751 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:08:07.751 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:08:07.751 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:08:08.893 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:08:08.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-0-256 2020-09-08T17:08:08.901 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:08:08.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:08:08.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-0-256 2020-09-08T17:08:08.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:08:08.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:08:08.902 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-size-59146-0-256 td/test-erasure-eio/COPY 2020-09-08T17:08:17.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:08:17.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:08:17.315 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: expr 0 + 1 2020-09-08T17:08:17.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: shard_id=1 2020-09-08T17:08:17.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: set_size obj-size-59146-0-256 td/test-erasure-eio 1 256 add 2020-09-08T17:08:17.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-0-256 2020-09-08T17:08:17.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:08:17.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:08:17.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:08:17.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=1 2020-09-08T17:08:17.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:08:17.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=256 2020-09-08T17:08:17.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:08:17.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=add 2020-09-08T17:08:17.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:08:17.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:08:17.319 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-0-256 2020-09-08T17:08:17.320 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:08:17.320 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-0-256 2020-09-08T17:08:17.320 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-0-256 2020-09-08T17:08:17.321 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:08:17.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:08:17.713 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:08:17.713 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:08:17.713 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:08:17.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:08:17.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=1 2020-09-08T17:08:17.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:08:18.364 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:08:18.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' add = add ']' 2020-09-08T17:08:18.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:18.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:08:18.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:08:18.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:08:18.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:08:18.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:18.375 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:08:18.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:08:18.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:08:18.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:08:18.376 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:08:18.376 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:08:18.377 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:08:18.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:08:18.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:08:18.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:08:18.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:08:18.479 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:18.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:08:18.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:08:18.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:08:18.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:08:18.480 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:08:18.481 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-0-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:20.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:08:20.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:08:20.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:08:20.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:08:20.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:08:20.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:08:20.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:08:20.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:08:20.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:08:20.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:08:20.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:08:20.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:08:20.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:08:20.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:08:20.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:08:20.611 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:08:20.611 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:20.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:20.612 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:20.612 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:20.612 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:20.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:20.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:20.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:08:20.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:08:20.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:08:20.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:08:20.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:08:20.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:20.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:08:20.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:08:20.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:08:20.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:08:20.616 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:08:20.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:20.616 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:08:20.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:08:20.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:08:20.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:08:20.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:08:20.630 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:20.637+0000 7fc02f658ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:20.632 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:20.639+0000 7fc02f658ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:20.634 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:20.641+0000 7fc02f658ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:20.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:08:20.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:08:20.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:08:20.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:08:20.989 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:08:20.989 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:20.989 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:08:20.989 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:08:20.989 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:20.990 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:08:21.156 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:21.163+0000 7fc02f658ec0 -1 Falling back to public interface 2020-09-08T17:08:21.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:08:22.096 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:22.103+0000 7fc02f658ec0 -1 osd.1 85 log_to_monitors {default=true} 2020-09-08T17:08:22.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:08:22.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:22.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:08:22.353 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:08:22.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:22.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:08:22.739 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 88 last_clean_interval [68,85) [v2:127.0.0.1:6810/151466,v1:127.0.0.1:6811/151466] [v2:127.0.0.1:6812/151466,v1:127.0.0.1:6813/151466] exists,up 285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:08:22.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:08:22.739 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:08:22.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:08:22.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:08:22.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:08:22.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:08:22.740 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:08:22.741 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:08:22.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:08:22.741 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:08:22.741 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:08:22.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:08:22.741 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:08:22.742 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:08:22.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:08:22.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:08:22.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:08:22.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:08:22.831 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:08:23.200 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:08:23.200 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:08:23.201 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:08:23.201 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:08:23.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:08:23.201 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:23.201 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:08:23.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803818 2020-09-08T17:08:23.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803818 2020-09-08T17:08:23.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803818' 2020-09-08T17:08:23.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:23.321 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:08:23.441 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=390842023939 2020-09-08T17:08:23.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 390842023939 2020-09-08T17:08:23.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803818 1-390842023939' 2020-09-08T17:08:23.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:23.442 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:08:23.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411367 2020-09-08T17:08:23.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411367 2020-09-08T17:08:23.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803818 1-390842023939 2-77309411367' 2020-09-08T17:08:23.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:23.562 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:08:23.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318280 2020-09-08T17:08:23.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318280 2020-09-08T17:08:23.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803818 1-390842023939 2-77309411367 3-352187318280' 2020-09-08T17:08:23.691 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:23.691 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803818 2020-09-08T17:08:23.692 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:23.692 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:08:23.693 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803818 2020-09-08T17:08:23.693 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:23.693 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803818 2020-09-08T17:08:23.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2020-09-08T17:08:23.694 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803818 2020-09-08T17:08:23.694 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:08:24.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803818 -lt 25769803818 2020-09-08T17:08:24.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:24.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-390842023939 2020-09-08T17:08:24.059 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:24.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:08:24.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-390842023939 2020-09-08T17:08:24.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:24.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=390842023939 2020-09-08T17:08:24.061 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 390842023939' 2020-09-08T17:08:24.062 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 390842023939 2020-09-08T17:08:24.062 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:08:24.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 390842023939 -lt 390842023939 2020-09-08T17:08:24.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:24.436 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411367 2020-09-08T17:08:24.436 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:24.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:08:24.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411367 2020-09-08T17:08:24.437 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:24.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411367 2020-09-08T17:08:24.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411367' 2020-09-08T17:08:24.438 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411367 2020-09-08T17:08:24.439 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:08:24.803 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411367 -lt 77309411367 2020-09-08T17:08:24.803 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:24.804 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-352187318280 2020-09-08T17:08:24.804 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:24.804 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:08:24.805 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-352187318280 2020-09-08T17:08:24.805 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:24.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318280 2020-09-08T17:08:24.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 352187318280' 2020-09-08T17:08:24.806 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 352187318280 2020-09-08T17:08:24.806 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:08:25.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318278 -lt 352187318280 2020-09-08T17:08:25.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:08:26.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:08:26.171 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:08:26.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318281 -lt 352187318280 2020-09-08T17:08:26.545 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:08:26.546 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:08:26.546 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:08:27.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:08:27.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:08:27.024 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:08:27.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:08:27.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:08:27.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:08:27.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:08:27.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:08:27.377 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:08:27.377 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:08:27.378 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:08:27.378 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:08:27.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:08:27.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:08:27.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:08:27.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: dd if=/dev/urandom bs=256 count=1 2020-09-08T17:08:27.864 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:08:27.865 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:08:27.865 INFO:tasks.workunit.client.0.smithi120.stderr:256 bytes copied, 4.4711e-05 s, 5.7 MB/s 2020-09-08T17:08:27.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:27.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:08:27.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:08:27.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:08:27.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:08:27.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:27.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:08:27.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:08:27.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:08:27.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:08:27.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:08:27.867 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:08:27.867 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:08:27.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:08:27.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:08:27.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:08:27.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:08:27.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:27.967 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:08:27.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:08:27.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:08:27.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:08:27.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:08:27.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-0-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:08:30.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:08:30.106 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:08:30.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:08:30.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:08:30.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:08:30.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:08:30.107 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:08:30.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:08:30.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:08:30.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:08:30.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:08:30.108 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:08:30.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:08:30.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:08:30.109 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:08:30.109 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:08:30.109 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:30.110 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:30.110 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:30.110 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:30.110 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:30.110 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:30.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:30.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:08:30.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:08:30.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:08:30.111 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:08:30.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:08:30.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:30.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:08:30.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:08:30.112 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:08:30.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:08:30.114 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:08:30.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:30.114 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:08:30.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:08:30.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:08:30.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:08:30.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:08:30.132 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:30.135+0000 7f598b21aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:30.132 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:30.137+0000 7f598b21aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:30.132 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:30.139+0000 7f598b21aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:30.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:08:30.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:08:30.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:08:30.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:08:30.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:08:30.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:30.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:08:30.503 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:08:30.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:08:30.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:30.656 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:30.662+0000 7f598b21aec0 -1 Falling back to public interface 2020-09-08T17:08:30.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:08:31.535 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:31.542+0000 7f598b21aec0 -1 osd.1 93 log_to_monitors {default=true} 2020-09-08T17:08:31.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:08:31.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:31.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:08:31.868 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:08:31.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:31.868 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:08:32.366 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 97 up_thru 97 down_at 94 last_clean_interval [91,93) [v2:127.0.0.1:6810/152589,v1:127.0.0.1:6811/152589] [v2:127.0.0.1:6812/152589,v1:127.0.0.1:6813/152589] exists,up 285af751-9c8b-4712-a97b-8e52a98e090e 2020-09-08T17:08:32.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:08:32.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:08:32.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:08:32.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:08:32.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:08:32.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:08:32.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:08:32.368 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:08:32.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:08:32.368 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:08:32.368 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:08:32.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:08:32.369 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:08:32.369 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:08:32.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:08:32.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:08:32.457 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:08:32.458 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:08:32.458 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:08:32.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:08:32.824 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:08:32.825 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:08:32.825 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:08:32.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:08:32.825 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:32.825 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:08:32.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803822 2020-09-08T17:08:32.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803822 2020-09-08T17:08:32.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803822' 2020-09-08T17:08:32.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:32.945 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:08:33.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=416611827715 2020-09-08T17:08:33.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 416611827715 2020-09-08T17:08:33.065 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803822 1-416611827715' 2020-09-08T17:08:33.066 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:33.066 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:08:33.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411371 2020-09-08T17:08:33.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411371 2020-09-08T17:08:33.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803822 1-416611827715 2-77309411371' 2020-09-08T17:08:33.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:08:33.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:08:33.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318284 2020-09-08T17:08:33.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318284 2020-09-08T17:08:33.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803822 1-416611827715 2-77309411371 3-352187318284' 2020-09-08T17:08:33.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:33.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803822 2020-09-08T17:08:33.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:33.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:08:33.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803822 2020-09-08T17:08:33.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:33.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803822 2020-09-08T17:08:33.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2020-09-08T17:08:33.306 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803822 2020-09-08T17:08:33.307 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:08:33.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803820 -lt 25769803822 2020-09-08T17:08:33.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:08:34.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:08:34.666 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:08:35.046 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803822 -lt 25769803822 2020-09-08T17:08:35.046 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:35.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-416611827715 2020-09-08T17:08:35.047 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:35.047 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:08:35.048 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-416611827715 2020-09-08T17:08:35.048 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:35.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=416611827715 2020-09-08T17:08:35.049 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 416611827715' 2020-09-08T17:08:35.049 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 416611827715 2020-09-08T17:08:35.050 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:08:35.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 416611827715 -lt 416611827715 2020-09-08T17:08:35.412 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:35.412 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411371 2020-09-08T17:08:35.413 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:35.413 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:08:35.414 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411371 2020-09-08T17:08:35.414 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:35.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411371 2020-09-08T17:08:35.415 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411371' 2020-09-08T17:08:35.415 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411371 2020-09-08T17:08:35.415 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:08:35.777 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411372 -lt 77309411371 2020-09-08T17:08:35.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:08:35.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-352187318284 2020-09-08T17:08:35.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:08:35.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:08:35.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-352187318284 2020-09-08T17:08:35.780 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:08:35.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318284 2020-09-08T17:08:35.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 352187318284' 2020-09-08T17:08:35.781 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 352187318284 2020-09-08T17:08:35.781 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:08:36.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318285 -lt 352187318284 2020-09-08T17:08:36.140 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:08:36.141 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:08:36.141 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:08:36.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:08:36.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:08:36.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:08:36.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:08:36.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:08:36.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:08:36.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:08:36.626 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:08:36.976 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:08:36.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:08:36.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:08:36.977 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:08:37.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:08:37.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:08:37.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:08:37.459 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:08:37.460 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:08:38.266 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:08:38.274 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-0-256 fail 2020-09-08T17:08:38.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:08:38.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:08:38.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-0-256 2020-09-08T17:08:38.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:08:38.275 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:08:38.276 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-size-59146-0-256 td/test-erasure-eio/COPY 2020-09-08T17:08:38.423 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-size-59146-0-256: (5) Input/output error 2020-09-08T17:08:38.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:08:38.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:08:38.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:08:38.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:08:38.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:08:38.943 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:08:38.950 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:08:39.506 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:08:39.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:08:39.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:08:39.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:08:39.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:08:39.516 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:08:39.516 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:08:39.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:08:39.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:08:39.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:08:39.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:08:39.627 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:08:39.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:08:39.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:08:39.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:08:39.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:08:39.629 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:08:39.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:08:39.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:08:39.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:08:39.632 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:08:39.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:08:39.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:08:39.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:08:39.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:08:39.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:39.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:39.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:08:39.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:08:39.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:08:39.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:08:39.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:08:39.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:08:39.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:08:39.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:08:39.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:08:39.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:08:39.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:08:39.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:08:39.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:08:39.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:08:39.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:08:39.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:08:39.655 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:08:39.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:08:39.659 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:08:39.659 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:08:39.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:08:39.660 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:08:39.660 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:08:39.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:08:39.661 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:08:39.661 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:08:39.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:08:39.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:08:39.662 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:08:39.663 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:08:39.663 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:39.663 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:39.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:08:39.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:08:39.664 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:08:39.665 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:08:39.665 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:08:39.665 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:39.666 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:39.666 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:08:39.669 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:08:39.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:08:39.669 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:08:39.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:08:39.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:08:39.670 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:08:39.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:08:39.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:08:39.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:08:39.671 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:08:39.672 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:08:39.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:08:39.966 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:39.967 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:39.967 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:39.967 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:39.967 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:39.968 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:39.968 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:08:40.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:08:40.126 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:08:40.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:08:40.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:08:40.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:08:40.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:08:40.127 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:08:40.128 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:08:40.128 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:08:40.128 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:08:40.128 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:40.128 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:40.129 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:08:40.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:08:40.130 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:08:40.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:08:40.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:08:40.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:08:40.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:08:40.216 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:08:40.216 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:08:40.216 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:08:40.216 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:08:40.216 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:08:40.217 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:40.217 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:40.217 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:08:40.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:08:40.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:08:40.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:08:40.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:08:40.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:08:40.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:08:40.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:08:40.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:08:40.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:08:41.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:08:41.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:41.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:41.186 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:41.187 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:41.187 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:41.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:41.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:08:41.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:08:41.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:08:41.208 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:08:41.488 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:08:41.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:08:42.498 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:08:42.498 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:08:42.498 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:08:42.499 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:08:42.499 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:42.499 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:42.499 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:08:42.499 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:08:42.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:08:42.579 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:08:42.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:08:42.588 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:08:40.001+0000 7f43b97cc6c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:08:42.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2020-09-08T17:08:42.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:315: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2020-09-08T17:08:42.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: setup_osds 4 2020-09-08T17:08:42.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=4 2020-09-08T17:08:42.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:08:42.593 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 4 - 1 2020-09-08T17:08:42.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 3 2020-09-08T17:08:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:08:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:08:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:08:42.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:08:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:08:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:08:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:08:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:08:42.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:08:42.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:08:42.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:08:42.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:08:42.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:08:42.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:08:42.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:08:42.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:08:42.597 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:08:42.597 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:42.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:42.598 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:42.598 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:42.598 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:42.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:42.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:42.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:08:42.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:08:42.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:08:42.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:08:42.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:08:42.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:08:42.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:08:42.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:42.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:08:42.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:08:42.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:08:42.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:08:42.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d760ead6-09fb-4f7c-9b52-a51f32cc88a2 2020-09-08T17:08:42.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d760ead6-09fb-4f7c-9b52-a51f32cc88a2' 2020-09-08T17:08:42.602 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:08:42.603 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 d760ead6-09fb-4f7c-9b52-a51f32cc88a2 2020-09-08T17:08:42.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCauldfDfKXJBAALNkdVTTDwHM8/oI8WhzdGg== 2020-09-08T17:08:42.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCauldfDfKXJBAALNkdVTTDwHM8/oI8WhzdGg=="}' 2020-09-08T17:08:42.608 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d760ead6-09fb-4f7c-9b52-a51f32cc88a2 -i td/test-erasure-eio/0/new.json 2020-09-08T17:08:42.929 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:08:42.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:08:42.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCauldfDfKXJBAALNkdVTTDwHM8/oI8WhzdGg== --osd-uuid d760ead6-09fb-4f7c-9b52-a51f32cc88a2 2020-09-08T17:08:42.960 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:42.966+0000 7fa3c2cfcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:42.962 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:42.968+0000 7fa3c2cfcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:42.963 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:42.969+0000 7fa3c2cfcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:42.964 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:42.970+0000 7fa3c2cfcec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:08:42.964 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:42.970+0000 7fa3c2cfcec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:08:42.965 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:42.970+0000 7fa3c2cfcec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:08:43.022 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:43.028+0000 7fa3c2cfcec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:08:46.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:08:46.986 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:08:46.987 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:08:46.987 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:08:46.988 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:08:47.487 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:08:47.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:08:47.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:47.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:08:47.488 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:08:47.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:08:47.506 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:47.512+0000 7ffa5fcbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:47.508 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:47.514+0000 7ffa5fcbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:47.509 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:47.515+0000 7ffa5fcbfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:47.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:08:47.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:08:47.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:08:47.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:08:47.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:08:47.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:47.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:08:47.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:47.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:08:47.854 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:08:48.024 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:48.030+0000 7ffa5fcbfec0 -1 Falling back to public interface 2020-09-08T17:08:48.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:08:48.526 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:48.532+0000 7ffa5fcbfec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:08:49.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:08:49.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:49.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:08:49.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:49.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:08:49.220 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:08:49.706 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/154446,v1:127.0.0.1:6803/154446] [v2:127.0.0.1:6804/154446,v1:127.0.0.1:6805/154446] exists,up d760ead6-09fb-4f7c-9b52-a51f32cc88a2 2020-09-08T17:08:49.707 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:08:49.707 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:08:49.707 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:08:49.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:08:49.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:08:49.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:08:49.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:08:49.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:08:49.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:08:49.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:08:49.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:08:49.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:08:49.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:08:49.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:08:49.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:08:49.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:08:49.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:08:49.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:08:49.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:08:49.711 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:08:49.711 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:08:49.711 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:49.711 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:49.711 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:49.712 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:49.712 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:49.712 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:49.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:49.712 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:08:49.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:08:49.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:08:49.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:08:49.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:08:49.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:08:49.713 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:08:49.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:49.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:08:49.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:08:49.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:08:49.714 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:08:49.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:08:49.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 560e2dea-86b2-4218-b715-99f7a291333f' 2020-09-08T17:08:49.715 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:08:49.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQChuldfdlBGKxAAFcMf/XrEx7pEY9kbo4OMXw== 2020-09-08T17:08:49.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQChuldfdlBGKxAAFcMf/XrEx7pEY9kbo4OMXw=="}' 2020-09-08T17:08:49.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 560e2dea-86b2-4218-b715-99f7a291333f -i td/test-erasure-eio/1/new.json 2020-09-08T17:08:50.195 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:08:50.205 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:08:50.206 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQChuldfdlBGKxAAFcMf/XrEx7pEY9kbo4OMXw== --osd-uuid 560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:08:50.227 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:50.232+0000 7fef7f7fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:50.228 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:50.234+0000 7fef7f7fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:50.229 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:50.235+0000 7fef7f7fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:50.230 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:50.236+0000 7fef7f7fcec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:08:50.230 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:50.236+0000 7fef7f7fcec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:08:50.230 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:50.236+0000 7fef7f7fcec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:08:50.275 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:50.281+0000 7fef7f7fcec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:08:54.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:08:54.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:08:54.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:08:54.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:08:54.263 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:08:54.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:08:54.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:54.763 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:08:54.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:08:54.764 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:08:54.764 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:08:54.782 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:54.788+0000 7f8e9852aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:54.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:54.789+0000 7f8e9852aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:54.785 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:54.791+0000 7f8e9852aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:55.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:08:55.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:08:55.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:08:55.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:08:55.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:08:55.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:55.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:08:55.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:55.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:08:55.131 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:08:55.303 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:55.309+0000 7f8e9852aec0 -1 Falling back to public interface 2020-09-08T17:08:55.492 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:08:56.078 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:56.084+0000 7f8e9852aec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:08:56.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:08:56.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:08:56.494 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:08:56.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:08:56.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:08:56.495 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:08:56.914 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/155440,v1:127.0.0.1:6811/155440] [v2:127.0.0.1:6812/155440,v1:127.0.0.1:6813/155440] exists,up 560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:08:56.914 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:08:56.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:08:56.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:08:56.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:08:56.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:08:56.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:08:56.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:08:56.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:08:56.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:08:56.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:08:56.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:08:56.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:08:56.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:08:56.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:08:56.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:08:56.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:08:56.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:08:56.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:08:56.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:08:56.918 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:08:56.918 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:08:56.918 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:08:56.919 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:08:56.919 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:08:56.919 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:08:56.919 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:08:56.919 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:56.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:08:56.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:08:56.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:08:56.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:08:56.920 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:08:56.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:08:56.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:08:56.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:08:56.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:08:56.921 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:08:56.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:08:56.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:08:56.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:08:56.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:08:56.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 b6975801-1d30-4729-a6a6-69d8802ecfb7' 2020-09-08T17:08:56.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:08:56.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCouldf2XCcNxAAA5SDkvmpt92g715bBanNWw== 2020-09-08T17:08:56.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCouldf2XCcNxAAA5SDkvmpt92g715bBanNWw=="}' 2020-09-08T17:08:56.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b6975801-1d30-4729-a6a6-69d8802ecfb7 -i td/test-erasure-eio/2/new.json 2020-09-08T17:08:57.586 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:08:57.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:08:57.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCouldf2XCcNxAAA5SDkvmpt92g715bBanNWw== --osd-uuid b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:08:57.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:57.624+0000 7fcf5c5cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:57.619 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:57.625+0000 7fcf5c5cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:57.620 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:57.626+0000 7fcf5c5cfec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:08:57.621 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:57.627+0000 7fcf5c5cfec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:08:57.621 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:57.627+0000 7fcf5c5cfec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:08:57.622 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:57.627+0000 7fcf5c5cfec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:08:57.706 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:08:57.713+0000 7fcf5c5cfec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:09:01.656 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:09:01.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:09:01.657 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:09:01.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:09:01.658 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:09:02.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:09:02.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:09:02.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:09:02.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:09:02.151 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:09:02.151 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:09:02.168 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:02.175+0000 7f6b9428fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:09:02.170 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:02.177+0000 7f6b9428fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:09:02.172 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:02.179+0000 7f6b9428fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:09:02.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:09:02.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:09:02.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:09:02.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:09:02.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:09:02.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:09:02.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:09:02.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:09:02.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:09:02.517 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:09:02.691 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:02.698+0000 7f6b9428fec0 -1 Falling back to public interface 2020-09-08T17:09:02.910 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:09:03.498 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:03.504+0000 7f6b9428fec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:09:03.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:09:03.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:09:03.912 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:09:03.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:09:03.913 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:09:03.913 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:09:04.280 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/156434,v1:127.0.0.1:6819/156434] [v2:127.0.0.1:6820/156434,v1:127.0.0.1:6821/156434] exists,up b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:09:04.281 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 a94c9366-ab85-481c-b162-87ab32865db8 2020-09-08T17:09:04.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:09:04.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:09:04.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:09:04.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:09:04.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:09:04.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:09:04.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:09:04.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:09:04.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:09:04.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:09:04.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:09:04.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:09:04.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:09:04.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:09:04.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:09:04.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:09:04.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:09:04.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:09:04.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:09:04.285 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:09:04.285 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:09:04.285 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:09:04.286 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:09:04.286 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:05.298 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:05.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:05.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:05.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:05.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:10:05.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:10:05.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:05.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:05.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:05.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:05.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:05.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:10:05.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:10:05.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:10:05.505 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:10:05.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a94c9366-ab85-481c-b162-87ab32865db8 2020-09-08T17:10:05.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 a94c9366-ab85-481c-b162-87ab32865db8' 2020-09-08T17:10:05.604 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:10:05.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCwuldf0hLlERAAm0D+qI3ket9Ha+wG04kumg== 2020-09-08T17:10:05.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCwuldf0hLlERAAm0D+qI3ket9Ha+wG04kumg=="}' 2020-09-08T17:10:05.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a94c9366-ab85-481c-b162-87ab32865db8 -i td/test-erasure-eio/3/new.json 2020-09-08T17:10:05.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:10:05.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCwuldf0hLlERAAm0D+qI3ket9Ha+wG04kumg== --osd-uuid a94c9366-ab85-481c-b162-87ab32865db8 2020-09-08T17:10:05.606 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:05.282+0000 7f4056cabec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:05.611 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:05.316+0000 7f4056cabec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:05.612 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:05.316+0000 7f4056cabec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:05.612 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:05.317+0000 7f4056cabec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:10:05.613 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:05.317+0000 7f4056cabec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:10:05.613 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:05.317+0000 7f4056cabec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:10:05.613 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:05.378+0000 7f4056cabec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:10:05.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:10:05.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:10:05.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:10:05.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:10:05.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:10:05.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:05.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:10:05.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:10:05.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:10:05.615 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:10.383+0000 7f04eeb5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:05.615 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:10.385+0000 7f04eeb5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:05.616 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:10.387+0000 7f04eeb5fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:05.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:10:05.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:05.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:10:05.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:05.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:05.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:05.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:05.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:05.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:10:05.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:10.906+0000 7f04eeb5fec0 -1 Falling back to public interface 2020-09-08T17:10:05.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:05.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:11.665+0000 7f04eeb5fec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:10:05.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:05.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:05.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:05.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:05.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:10:05.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:05.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:05.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:05.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:10:05.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:10:05.620 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:10:05.621 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:10:05.621 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:05.621 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:05.621 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:10:05.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:10:05.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:10:05.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:10:05.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:318: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2020-09-08T17:10:05.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2020-09-08T17:10:05.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:10:05.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:10:05.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=2 2020-09-08T17:10:05.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:10:05.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=1 2020-09-08T17:10:05.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:10:05.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2020-09-08T17:10:05.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:10:05.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:10:05.624 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:10:05.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:10:05.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:10:05.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:05.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:05.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:05.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:05.626 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:05.626 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:05.626 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:05.626 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:05.627 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:05.627 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:05.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:05.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:05.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:05.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:05.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:05.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:05.628 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:05.628 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:05.629 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:05.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:05.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:05.629 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:05.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803784 2020-09-08T17:10:05.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803784 2020-09-08T17:10:05.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784' 2020-09-08T17:10:05.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:05.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:05.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574854 2020-09-08T17:10:05.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574854 2020-09-08T17:10:05.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-55834574854' 2020-09-08T17:10:05.631 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:05.631 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:05.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378630 2020-09-08T17:10:05.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378630 2020-09-08T17:10:05.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-55834574854 2-81604378630' 2020-09-08T17:10:05.632 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:05.632 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:05.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149700 2020-09-08T17:10:05.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149700 2020-09-08T17:10:05.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-55834574854 2-81604378630 3-111669149700' 2020-09-08T17:10:05.633 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:05.633 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803784 2020-09-08T17:10:05.634 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:05.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:05.634 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803784 2020-09-08T17:10:05.634 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:05.634 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803784 2020-09-08T17:10:05.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2020-09-08T17:10:05.635 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:05.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803781 -lt 25769803784 2020-09-08T17:10:05.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:10:05.635 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:10:05.636 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:05.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803784 -lt 25769803784 2020-09-08T17:10:05.636 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:05.636 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574854 2020-09-08T17:10:05.636 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:05.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:05.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574854 2020-09-08T17:10:05.637 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:05.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574854 2020-09-08T17:10:05.637 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574854' 2020-09-08T17:10:05.638 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:05.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574854 -lt 55834574854 2020-09-08T17:10:05.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:05.638 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378630 2020-09-08T17:10:05.638 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:05.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:05.639 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378630 2020-09-08T17:10:05.639 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:05.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378630 2020-09-08T17:10:05.639 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378630' 2020-09-08T17:10:05.640 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:05.640 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378630 -lt 81604378630 2020-09-08T17:10:05.640 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:05.640 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149700 2020-09-08T17:10:05.640 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:05.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:05.641 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149700 2020-09-08T17:10:05.641 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:05.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149700 2020-09-08T17:10:05.641 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149700' 2020-09-08T17:10:05.642 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:05.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149701 -lt 111669149700 2020-09-08T17:10:05.642 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:05.642 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:05.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:05.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:05.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:05.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:05.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:05.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:05.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:05.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:05.847 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:05.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:05.848 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:05.848 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:05.848 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:05.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:05.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:05.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:05.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:321: TEST_rados_get_bad_size_shard_1: local shard_id=1 2020-09-08T17:10:05.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2020-09-08T17:10:05.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:221: rados_get_data_bad_size: local dir=td/test-erasure-eio 2020-09-08T17:10:05.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: shift 2020-09-08T17:10:05.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: local shard_id=1 2020-09-08T17:10:05.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: shift 2020-09-08T17:10:05.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: local bytes=10 2020-09-08T17:10:05.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: shift 2020-09-08T17:10:05.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: local mode=set 2020-09-08T17:10:05.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:229: rados_get_data_bad_size: local poolname=pool-jerasure 2020-09-08T17:10:05.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local objname=obj-size-59146-1-10 2020-09-08T17:10:05.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-59146-1-10 2020-09-08T17:10:05.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:10:05.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:10:05.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-size-59146-1-10 2020-09-08T17:10:05.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:05.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:10:05.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:05.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:10:05.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:05.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:10:05.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:05.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:10:05.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-size-59146-1-10 td/test-erasure-eio/ORIGINAL 2020-09-08T17:10:05.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: set_size obj-size-59146-1-10 td/test-erasure-eio 1 10 set 2020-09-08T17:10:06.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-1-10 2020-09-08T17:10:06.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:10:06.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:10:06.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:10:06.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=1 2020-09-08T17:10:06.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:10:06.059 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=10 2020-09-08T17:10:06.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:10:06.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:10:06.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:10:06.060 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:10:06.060 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-1-10 2020-09-08T17:10:06.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:10:06.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-1-10 2020-09-08T17:10:06.061 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-1-10 2020-09-08T17:10:06.061 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:10:06.061 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:10:06.062 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:06.062 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:10:06.062 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:10:06.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:10:06.062 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=1 2020-09-08T17:10:06.063 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:10:06.266 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:10:08.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:10:08.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 10 = 0 ']' 2020-09-08T17:10:08.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.315 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:10:08.315 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:10:08.315 INFO:tasks.workunit.client.0.smithi120.stderr:10 bytes copied, 4.0909e-05 s, 244 kB/s 2020-09-08T17:10:08.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.315 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:08.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:08.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:10:08.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:08.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.316 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:08.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:08.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:10:08.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:08.317 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:10:08.317 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:08.318 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:08.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:08.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:08.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:08.318 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:08.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:08.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:08.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:10:08.319 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:08.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:08.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:10:08.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:08.320 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:08.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:10:08.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:08.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:08.321 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:08.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:08.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:08.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:08.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:10:08.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:10:08.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:08.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:08.323 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:08.323 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:08.324 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:08.324 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:08.324 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:08.324 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:08.324 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:08.325 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:08.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:08.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:08.325 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:08.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:08.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:08.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:08.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:08.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:08.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:08.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:10:08.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:10:08.327 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:10:08.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:08.328 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:10:08.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:10:08.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:08.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:08.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:08.329 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:23.824+0000 7f1581655ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:08.329 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:23.827+0000 7f1581655ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:08.329 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:23.828+0000 7f1581655ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:08.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:10:08.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:08.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:10:08.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:08.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:08.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:08.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:08.331 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:10:08.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:08.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:08.331 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:24.345+0000 7f1581655ec0 -1 Falling back to public interface 2020-09-08T17:10:08.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:08.332 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:25.259+0000 7f1581655ec0 -1 osd.1 32 log_to_monitors {default=true} 2020-09-08T17:10:08.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:08.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:08.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:08.333 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:08.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:08.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:08.333 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [13,32) [v2:127.0.0.1:6810/158678,v1:127.0.0.1:6811/158678] [v2:127.0.0.1:6812/158678,v1:127.0.0.1:6813/158678] exists,up 560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:10:08.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:08.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:08.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:08.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:10:08.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:08.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:08.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:08.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:08.335 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:08.335 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:08.335 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:08.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:08.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:08.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:08.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:08.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:08.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:08.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:08.337 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:08.337 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:08.337 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:08.338 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:08.338 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:08.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:08.338 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.338 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:08.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803788 2020-09-08T17:10:08.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803788 2020-09-08T17:10:08.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788' 2020-09-08T17:10:08.339 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.339 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:08.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757251 2020-09-08T17:10:08.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757251 2020-09-08T17:10:08.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-163208757251' 2020-09-08T17:10:08.340 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.340 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:08.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378634 2020-09-08T17:10:08.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378634 2020-09-08T17:10:08.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-163208757251 2-81604378634' 2020-09-08T17:10:08.341 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.341 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:08.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149704 2020-09-08T17:10:08.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149704 2020-09-08T17:10:08.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-163208757251 2-81604378634 3-111669149704' 2020-09-08T17:10:08.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.342 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803788 2020-09-08T17:10:08.343 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:08.343 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803788 2020-09-08T17:10:08.343 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803788 2020-09-08T17:10:08.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2020-09-08T17:10:08.344 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803788 2020-09-08T17:10:08.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:08.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803788 2020-09-08T17:10:08.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-163208757251 2020-09-08T17:10:08.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:08.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-163208757251 2020-09-08T17:10:08.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757251 2020-09-08T17:10:08.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 163208757251' 2020-09-08T17:10:08.346 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 163208757251 2020-09-08T17:10:08.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:08.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 163208757251 2020-09-08T17:10:08.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:10:08.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:10:08.347 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:08.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757251 -lt 163208757251 2020-09-08T17:10:08.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378634 2020-09-08T17:10:08.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.348 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:08.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378634 2020-09-08T17:10:08.348 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378634 2020-09-08T17:10:08.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378634' 2020-09-08T17:10:08.349 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 81604378634 2020-09-08T17:10:08.349 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:08.349 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378634 -lt 81604378634 2020-09-08T17:10:08.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149704 2020-09-08T17:10:08.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.350 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:08.350 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149704 2020-09-08T17:10:08.351 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149704 2020-09-08T17:10:08.351 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149704' 2020-09-08T17:10:08.351 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149704 2020-09-08T17:10:08.351 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:08.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149705 -lt 111669149704 2020-09-08T17:10:08.352 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:08.352 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:08.352 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:08.352 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:08.353 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:08.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:08.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:08.353 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:08.354 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:08.355 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:08.355 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:08.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:08.355 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:08.355 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:08.356 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:08.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:08.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:08.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:08.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:10:08.357 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:10:08.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-1-10 2020-09-08T17:10:08.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:10:08.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:10:08.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-1-10 2020-09-08T17:10:08.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:10:08.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:10:08.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-size-59146-1-10 td/test-erasure-eio/COPY 2020-09-08T17:10:08.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:10:08.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:10:08.359 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: expr 1 + 1 2020-09-08T17:10:08.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: shard_id=2 2020-09-08T17:10:08.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: set_size obj-size-59146-1-10 td/test-erasure-eio 2 10 set 2020-09-08T17:10:08.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-1-10 2020-09-08T17:10:08.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:10:08.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:10:08.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:10:08.360 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=2 2020-09-08T17:10:08.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:10:08.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=10 2020-09-08T17:10:08.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:10:08.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:10:08.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:10:08.567 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:10:08.567 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-1-10 2020-09-08T17:10:08.567 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:10:08.568 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-1-10 2020-09-08T17:10:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-1-10 2020-09-08T17:10:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:10:08.568 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:10:08.568 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:10:08.569 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:10:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:10:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=2 2020-09-08T17:10:08.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:10:08.570 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:10:08.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:10:08.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 10 = 0 ']' 2020-09-08T17:10:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:213: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:10:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:10:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:10 bytes copied, 4.1547e-05 s, 241 kB/s 2020-09-08T17:10:08.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:08.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:08.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=2 2020-09-08T17:10:08.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:08.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:08.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:08.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2020-09-08T17:10:08.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:08.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2020-09-08T17:10:08.574 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:08.574 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:08.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:08.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:08.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:08.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:08.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:08.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:08.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2020-09-08T17:10:08.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:08.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:08.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-59146-1-10 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2020-09-08T17:10:08.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:08.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:08.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-09-08T17:10:08.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:08.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:08.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:08.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:08.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:08.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:08.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:10:08.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:10:08.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:08.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:08.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:08.579 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:08.579 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:08.580 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:08.580 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:08.580 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:08.580 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:08.580 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:08.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:08.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:08.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:08.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:08.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:08.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:08.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:08.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:08.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:08.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:10:08.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-09-08T17:10:08.587 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.2 2020-09-08T17:10:08.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:08.587 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/2/whoami 2020-09-08T17:10:08.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-09-08T17:10:08.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:08.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:08.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:08.588 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:35.580+0000 7f1898c5eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:08.589 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:35.583+0000 7f1898c5eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:08.589 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:35.584+0000 7f1898c5eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:08.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-09-08T17:10:08.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:08.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:10:08.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:08.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:08.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:08.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:08.590 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:10:08.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:08.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:10:08.591 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:36.099+0000 7f1898c5eec0 -1 Falling back to public interface 2020-09-08T17:10:08.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:37.032+0000 7f1898c5eec0 -1 osd.2 42 log_to_monitors {default=true} 2020-09-08T17:10:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:08.592 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:10:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 up in weight 1 up_from 48 up_thru 35 down_at 45 last_clean_interval [19,42) [v2:127.0.0.1:6818/159908,v1:127.0.0.1:6819/159908] [v2:127.0.0.1:6820/159908,v1:127.0.0.1:6821/159908] exists,up b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:10:08.593 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:08.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:08.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:08.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:10:08.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:08.594 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:08.595 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:08.595 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:08.596 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:08.596 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:08.596 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:08.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:08.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:08.597 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:08.597 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:08.598 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:08.806 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:08.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:08.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.807 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:08.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803792 2020-09-08T17:10:08.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803792 2020-09-08T17:10:08.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792' 2020-09-08T17:10:08.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:08.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757255 2020-09-08T17:10:08.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757255 2020-09-08T17:10:08.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-163208757255' 2020-09-08T17:10:08.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:08.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=206158430211 2020-09-08T17:10:08.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 206158430211 2020-09-08T17:10:08.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-163208757255 2-206158430211' 2020-09-08T17:10:08.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:08.810 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:08.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149709 2020-09-08T17:10:08.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149709 2020-09-08T17:10:08.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-163208757255 2-206158430211 3-111669149709' 2020-09-08T17:10:08.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803792 2020-09-08T17:10:08.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.811 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:08.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803792 2020-09-08T17:10:08.811 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803792 2020-09-08T17:10:08.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803792' 2020-09-08T17:10:08.812 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803792 2020-09-08T17:10:08.812 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:08.812 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803792 -lt 25769803792 2020-09-08T17:10:08.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.813 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-163208757255 2020-09-08T17:10:08.813 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.813 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:08.813 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-163208757255 2020-09-08T17:10:08.814 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757255 2020-09-08T17:10:08.814 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 163208757255' 2020-09-08T17:10:08.814 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 163208757255 2020-09-08T17:10:08.814 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:08.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757255 -lt 163208757255 2020-09-08T17:10:08.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.815 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-206158430211 2020-09-08T17:10:08.815 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.815 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:08.816 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-206158430211 2020-09-08T17:10:08.816 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=206158430211 2020-09-08T17:10:08.816 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2020-09-08T17:10:08.816 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 206158430211 2020-09-08T17:10:08.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:08.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 206158430211 -lt 206158430211 2020-09-08T17:10:08.817 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:08.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149709 2020-09-08T17:10:08.817 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:08.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:08.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149709 2020-09-08T17:10:08.818 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:08.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149709 2020-09-08T17:10:08.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149709' 2020-09-08T17:10:08.819 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149709 2020-09-08T17:10:08.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:08.819 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149709 -lt 111669149709 2020-09-08T17:10:08.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:08.819 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:08.820 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:08.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:08.820 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:08.820 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:08.820 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:08.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:08.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:08.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:08.821 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:08.821 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:08.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:08.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:08.822 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:08.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:08.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:08.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:08.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:10:08.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:10:08.823 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:10:08.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-1-10 fail 2020-09-08T17:10:08.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:10:08.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:10:08.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-1-10 2020-09-08T17:10:08.824 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:10:09.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:10:09.026 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-size-59146-1-10 td/test-erasure-eio/COPY 2020-09-08T17:10:09.026 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-size-59146-1-10: (5) Input/output error 2020-09-08T17:10:09.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:10:09.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:10:09.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2020-09-08T17:10:09.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:221: rados_get_data_bad_size: local dir=td/test-erasure-eio 2020-09-08T17:10:09.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: shift 2020-09-08T17:10:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: local shard_id=1 2020-09-08T17:10:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: shift 2020-09-08T17:10:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: local bytes=0 2020-09-08T17:10:09.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: shift 2020-09-08T17:10:09.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: local mode=set 2020-09-08T17:10:09.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:229: rados_get_data_bad_size: local poolname=pool-jerasure 2020-09-08T17:10:09.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local objname=obj-size-59146-1-0 2020-09-08T17:10:09.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-59146-1-0 2020-09-08T17:10:09.029 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:10:09.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:10:09.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-size-59146-1-0 2020-09-08T17:10:09.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:09.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:10:09.030 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:09.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:10:09.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:09.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:10:09.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:09.031 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:10:09.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-size-59146-1-0 td/test-erasure-eio/ORIGINAL 2020-09-08T17:10:09.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: set_size obj-size-59146-1-0 td/test-erasure-eio 1 0 set 2020-09-08T17:10:09.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-1-0 2020-09-08T17:10:09.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:10:09.032 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:10:09.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:10:09.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=1 2020-09-08T17:10:09.033 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:10:09.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=0 2020-09-08T17:10:09.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:10:09.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:10:09.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:10:09.035 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:10:09.035 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-1-0 2020-09-08T17:10:09.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:10:09.646 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-1-0 2020-09-08T17:10:09.646 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-1-0 2020-09-08T17:10:09.646 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:10:09.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:10:09.647 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:09.647 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:10:09.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:10:09.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:10:09.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=1 2020-09-08T17:10:09.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:10:09.648 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:10:09.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:10:09.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 0 = 0 ']' 2020-09-08T17:10:09.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:211: set_size: touch td/test-erasure-eio/CORRUPT 2020-09-08T17:10:09.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:09.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:09.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:09.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:10:09.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:09.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:09.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:09.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:09.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:10:11.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:11.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:10:11.286 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:11.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:11.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:11.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:11.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:11.287 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:11.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:11.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:11.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:10:11.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:10:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:11.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:11.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:10:11.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:11.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:11.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:11.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:11.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:11.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:11.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:10:11.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:10:11.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:11.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:11.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:11.292 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:11.292 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:11.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:11.293 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:11.293 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:11.293 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:11.293 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:11.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:11.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:11.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:11.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:11.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:11.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:11.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:11.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:11.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:11.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:10:11.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:10:11.296 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:10:11.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:11.296 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:10:11.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:10:11.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:11.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:11.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:11.297 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:45.814+0000 7fe1bf86dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:11.298 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:45.817+0000 7fe1bf86dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:11.298 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:45.818+0000 7fe1bf86dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:11.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:10:11.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:11.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:10:11.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:11.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:11.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:11.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:11.299 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:10:11.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:11.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:11.300 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:46.334+0000 7fe1bf86dec0 -1 Falling back to public interface 2020-09-08T17:10:11.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:11.300 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:47.223+0000 7fe1bf86dec0 -1 osd.1 51 log_to_monitors {default=true} 2020-09-08T17:10:11.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:11.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:11.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:11.301 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:11.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:11.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:11.302 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 54 last_clean_interval [38,51) [v2:127.0.0.1:6810/161123,v1:127.0.0.1:6811/161123] [v2:127.0.0.1:6812/161123,v1:127.0.0.1:6813/161123] exists,up 560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:10:11.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:11.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:11.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:11.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:10:11.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:11.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:11.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:11.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:11.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:11.304 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:11.304 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:11.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:11.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:11.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:11.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:11.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:11.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:11.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:11.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:11.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:11.306 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:11.306 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:11.306 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:11.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:11.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:11.307 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:11.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803796 2020-09-08T17:10:11.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803796 2020-09-08T17:10:11.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803796' 2020-09-08T17:10:11.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:11.308 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:11.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=244813135875 2020-09-08T17:10:11.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 244813135875 2020-09-08T17:10:11.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803796 1-244813135875' 2020-09-08T17:10:11.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:11.309 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:11.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=206158430215 2020-09-08T17:10:11.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 206158430215 2020-09-08T17:10:11.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803796 1-244813135875 2-206158430215' 2020-09-08T17:10:11.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:11.310 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:11.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149713 2020-09-08T17:10:11.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149713 2020-09-08T17:10:11.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803796 1-244813135875 2-206158430215 3-111669149713' 2020-09-08T17:10:11.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:11.311 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803796 2020-09-08T17:10:11.311 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:11.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:11.312 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803796 2020-09-08T17:10:11.312 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:11.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803796 2020-09-08T17:10:11.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803796' 2020-09-08T17:10:11.313 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803796 2020-09-08T17:10:11.313 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:11.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803796 -lt 25769803796 2020-09-08T17:10:11.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:11.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-244813135875 2020-09-08T17:10:11.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:11.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:11.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-244813135875 2020-09-08T17:10:11.719 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:11.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=244813135875 2020-09-08T17:10:11.719 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 244813135875' 2020-09-08T17:10:11.719 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 244813135875 2020-09-08T17:10:11.720 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:11.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 244813135875 -lt 244813135875 2020-09-08T17:10:11.720 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:11.720 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-206158430215 2020-09-08T17:10:11.720 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:11.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:11.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-206158430215 2020-09-08T17:10:11.721 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:11.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=206158430215 2020-09-08T17:10:11.721 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 206158430215' 2020-09-08T17:10:11.722 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 206158430215 2020-09-08T17:10:11.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:11.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 206158430213 -lt 206158430215 2020-09-08T17:10:11.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:10:11.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:10:11.723 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:11.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 206158430215 -lt 206158430215 2020-09-08T17:10:11.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:11.723 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149713 2020-09-08T17:10:11.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:11.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:11.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149713 2020-09-08T17:10:11.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:11.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149713 2020-09-08T17:10:11.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149713' 2020-09-08T17:10:11.725 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149713 2020-09-08T17:10:11.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:11.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149713 -lt 111669149713 2020-09-08T17:10:11.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:11.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:11.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:11.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:11.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:11.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:11.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:11.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:11.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:11.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:11.727 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:11.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:11.728 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:11.728 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:11.728 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:11.728 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:11.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:11.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:11.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:10:11.729 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:10:11.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-1-0 2020-09-08T17:10:11.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:10:11.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:10:11.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-1-0 2020-09-08T17:10:11.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:10:11.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:10:11.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-size-59146-1-0 td/test-erasure-eio/COPY 2020-09-08T17:10:11.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:10:11.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:10:11.731 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: expr 1 + 1 2020-09-08T17:10:11.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: shard_id=2 2020-09-08T17:10:11.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: set_size obj-size-59146-1-0 td/test-erasure-eio 2 0 set 2020-09-08T17:10:11.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-1-0 2020-09-08T17:10:11.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:10:11.732 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:10:11.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:10:11.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=2 2020-09-08T17:10:11.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:10:11.733 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=0 2020-09-08T17:10:11.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:10:11.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=set 2020-09-08T17:10:11.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:10:11.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:10:11.734 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-1-0 2020-09-08T17:10:11.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:10:11.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-1-0 2020-09-08T17:10:11.735 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-1-0 2020-09-08T17:10:11.735 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:10:11.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:10:11.736 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:11.736 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:10:11.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:10:11.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:10:11.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=2 2020-09-08T17:10:11.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:10:11.939 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:10:11.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' set = add ']' 2020-09-08T17:10:11.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: '[' 0 = 0 ']' 2020-09-08T17:10:11.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:211: set_size: touch td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:11.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:11.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=2 2020-09-08T17:10:11.940 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:11.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:11.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:11.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2020-09-08T17:10:11.941 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:11.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2020-09-08T17:10:11.942 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:11.942 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:11.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:11.942 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:11.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:11.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:11.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:11.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:11.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2020-09-08T17:10:11.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:11.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:11.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-59146-1-0 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:11.944 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2020-09-08T17:10:11.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:11.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:11.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-09-08T17:10:11.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:11.945 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:11.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:11.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:11.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:11.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:11.946 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:10:12.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:10:12.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:12.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:12.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:12.563 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:12.563 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:12.563 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:22.174 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:22.175 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:22.175 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:22.175 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:22.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:22.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:22.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:22.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:22.176 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:22.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:22.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:22.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:22.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:22.177 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:10:22.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-09-08T17:10:22.178 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.2 2020-09-08T17:10:22.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:22.178 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/2/whoami 2020-09-08T17:10:22.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-09-08T17:10:22.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:22.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:22.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:22.180 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:57.024+0000 7fddc459bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:22.180 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:57.026+0000 7fddc459bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:22.180 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:57.028+0000 7fddc459bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:22.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-09-08T17:10:22.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:22.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:10:22.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:22.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:22.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:22.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:22.182 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:10:22.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:22.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:10:22.182 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:57.547+0000 7fddc459bec0 -1 Falling back to public interface 2020-09-08T17:10:22.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:22.183 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:09:58.390+0000 7fddc459bec0 -1 osd.2 62 log_to_monitors {default=true} 2020-09-08T17:10:22.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:22.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:22.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:22.184 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:22.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:22.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:10:22.184 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 up in weight 1 up_from 66 up_thru 54 down_at 64 last_clean_interval [48,62) [v2:127.0.0.1:6818/162345,v1:127.0.0.1:6819/162345] [v2:127.0.0.1:6820/162345,v1:127.0.0.1:6821/162345] exists,up b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:10:22.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:22.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:22.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:22.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:10:22.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:22.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:22.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:22.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:22.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:22.187 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:22.187 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:22.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:22.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:22.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:22.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:22.390 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:22.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:22.391 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:22.391 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:22.392 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:22.392 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:22.392 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:22.393 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:22.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:22.393 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:22.393 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803800 2020-09-08T17:10:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803800 2020-09-08T17:10:22.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803800' 2020-09-08T17:10:22.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:22.395 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:22.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=244813135879 2020-09-08T17:10:22.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 244813135879 2020-09-08T17:10:22.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803800 1-244813135879' 2020-09-08T17:10:22.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:22.396 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:22.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=283467841539 2020-09-08T17:10:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 283467841539 2020-09-08T17:10:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803800 1-244813135879 2-283467841539' 2020-09-08T17:10:22.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:22.397 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:22.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149717 2020-09-08T17:10:22.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149717 2020-09-08T17:10:22.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803800 1-244813135879 2-283467841539 3-111669149717' 2020-09-08T17:10:22.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:22.404 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803800 2020-09-08T17:10:22.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:22.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:22.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803800 2020-09-08T17:10:22.406 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:22.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803800 2020-09-08T17:10:22.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803800' 2020-09-08T17:10:22.407 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803800 2020-09-08T17:10:22.407 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803798 -lt 25769803800 2020-09-08T17:10:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:10:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:10:22.408 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:22.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803801 -lt 25769803800 2020-09-08T17:10:22.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:22.409 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-244813135879 2020-09-08T17:10:22.409 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:22.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:22.610 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-244813135879 2020-09-08T17:10:22.611 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:22.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=244813135879 2020-09-08T17:10:22.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2020-09-08T17:10:22.611 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 244813135879 2020-09-08T17:10:22.611 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:22.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 244813135879 -lt 244813135879 2020-09-08T17:10:22.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:22.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-283467841539 2020-09-08T17:10:22.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:22.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:22.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-283467841539 2020-09-08T17:10:22.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:22.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=283467841539 2020-09-08T17:10:22.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 283467841539' 2020-09-08T17:10:22.613 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 283467841539 2020-09-08T17:10:22.614 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:22.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 283467841539 -lt 283467841539 2020-09-08T17:10:22.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:22.614 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149717 2020-09-08T17:10:22.614 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:22.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:22.615 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149717 2020-09-08T17:10:22.615 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:22.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149717 2020-09-08T17:10:22.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149717' 2020-09-08T17:10:22.616 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149717 2020-09-08T17:10:22.616 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:22.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149717 -lt 111669149717 2020-09-08T17:10:22.616 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:22.616 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:22.617 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:22.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:22.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:22.617 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:22.618 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:22.618 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:22.618 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:22.618 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:22.618 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:22.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:22.619 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:22.619 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:22.619 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:22.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:22.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:22.822 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:22.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:10:22.823 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:10:23.842 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:10:23.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-1-0 fail 2020-09-08T17:10:23.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:10:23.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:10:23.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-1-0 2020-09-08T17:10:23.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:10:23.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:10:23.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-size-59146-1-0 td/test-erasure-eio/COPY 2020-09-08T17:10:23.844 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-size-59146-1-0: (5) Input/output error 2020-09-08T17:10:23.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:10:23.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:10:23.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:324: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2020-09-08T17:10:23.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:221: rados_get_data_bad_size: local dir=td/test-erasure-eio 2020-09-08T17:10:23.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: shift 2020-09-08T17:10:23.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: local shard_id=1 2020-09-08T17:10:23.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: shift 2020-09-08T17:10:23.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: local bytes=256 2020-09-08T17:10:23.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: shift 2020-09-08T17:10:23.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: local mode=add 2020-09-08T17:10:23.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:229: rados_get_data_bad_size: local poolname=pool-jerasure 2020-09-08T17:10:23.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local objname=obj-size-59146-1-256 2020-09-08T17:10:23.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-59146-1-256 2020-09-08T17:10:23.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:10:23.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:10:23.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-size-59146-1-256 2020-09-08T17:10:23.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:23.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:10:23.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:23.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:10:23.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:23.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:10:23.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:10:23.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:10:23.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-size-59146-1-256 td/test-erasure-eio/ORIGINAL 2020-09-08T17:10:23.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:235: rados_get_data_bad_size: set_size obj-size-59146-1-256 td/test-erasure-eio 1 256 add 2020-09-08T17:10:23.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-1-256 2020-09-08T17:10:23.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:10:23.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:10:23.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:10:23.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=1 2020-09-08T17:10:23.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:10:23.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=256 2020-09-08T17:10:23.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:10:23.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=add 2020-09-08T17:10:23.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:10:23.851 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:10:23.852 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-1-256 2020-09-08T17:10:23.852 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:10:23.852 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-1-256 2020-09-08T17:10:23.852 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-1-256 2020-09-08T17:10:23.852 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:10:23.853 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:10:23.853 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:10:23.854 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:10:23.854 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:10:23.854 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:10:23.854 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/157428,v1:127.0.0.1:6827/157428] [v2:127.0.0.1:6828/157428,v1:127.0.0.1:6829/157428] exists,up a94c9366-ab85-481c-b162-87ab32865db8 2020-09-08T17:10:23.854 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:10:23.855 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:08:48.036+0000 7ffa5fcbfec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:10:23.855 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803784 2020-09-08T17:10:23.855 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 55834574854 2020-09-08T17:10:23.855 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 81604378630 2020-09-08T17:10:24.062 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149700 2020-09-08T17:10:24.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:10:24.091 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:24.091 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:10:24.091 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:10:24.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:10:24.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=1 2020-09-08T17:10:24.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:10:24.092 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:10:24.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' add = add ']' 2020-09-08T17:10:24.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:24.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:24.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:24.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:10:24.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:24.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:24.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:24.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:24.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:10:24.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:24.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:10:24.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:24.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:24.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:24.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:24.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:24.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:24.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:24.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:24.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:24.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:10:24.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:24.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:24.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:24.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:10:24.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:24.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:24.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:10:24.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:24.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:24.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:24.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:24.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:24.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:24.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:10:24.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:10:24.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:24.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:24.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:24.303 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:24.303 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:24.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:24.304 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:24.304 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:24.304 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:24.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:24.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:24.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:24.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:24.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:24.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:24.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:24.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:24.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:24.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:24.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:10:24.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:10:24.307 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:10:24.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:24.307 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:10:24.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:10:24.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:24.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:24.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:24.308 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:08.462+0000 7f1de03edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:24.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:08.464+0000 7f1de03edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:24.309 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:08.465+0000 7f1de03edec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:24.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:10:24.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:24.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:10:24.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:24.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:24.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:24.310 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:24.310 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:10:24.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:24.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:24.311 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:08.982+0000 7f1de03edec0 -1 Falling back to public interface 2020-09-08T17:10:24.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:24.514 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:09.866+0000 7f1de03edec0 -1 osd.1 71 log_to_monitors {default=true} 2020-09-08T17:10:24.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:24.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:24.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:24.515 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:24.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:24.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:24.516 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 76 up_thru 76 down_at 73 last_clean_interval [57,71) [v2:127.0.0.1:6810/163588,v1:127.0.0.1:6811/163588] [v2:127.0.0.1:6812/163588,v1:127.0.0.1:6813/163588] exists,up 560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:10:24.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:24.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:24.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:24.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:10:24.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:24.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:24.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:24.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:24.517 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:24.518 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:24.518 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:24.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:24.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:24.519 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:24.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:24.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:24.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:24.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:24.927 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:24.927 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:24.927 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:24.927 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:24.927 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:24.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:24.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:24.928 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:24.928 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803805 2020-09-08T17:10:24.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803805 2020-09-08T17:10:24.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805' 2020-09-08T17:10:24.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:24.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:24.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=326417514499 2020-09-08T17:10:24.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 326417514499 2020-09-08T17:10:24.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-326417514499' 2020-09-08T17:10:24.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:24.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:24.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=283467841543 2020-09-08T17:10:24.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 283467841543 2020-09-08T17:10:24.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-326417514499 2-283467841543' 2020-09-08T17:10:25.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:25.134 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:25.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149721 2020-09-08T17:10:25.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149721 2020-09-08T17:10:25.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-326417514499 2-283467841543 3-111669149721' 2020-09-08T17:10:25.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:25.135 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803805 2020-09-08T17:10:25.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:25.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:25.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803805 2020-09-08T17:10:25.136 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:25.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803805 2020-09-08T17:10:25.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2020-09-08T17:10:25.137 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803805 2020-09-08T17:10:25.137 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:25.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803803 -lt 25769803805 2020-09-08T17:10:25.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:10:25.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:10:25.138 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:25.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803805 -lt 25769803805 2020-09-08T17:10:25.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:25.138 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-326417514499 2020-09-08T17:10:25.342 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:25.342 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:25.342 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-326417514499 2020-09-08T17:10:25.343 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:25.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=326417514499 2020-09-08T17:10:25.343 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2020-09-08T17:10:25.343 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 326417514499 2020-09-08T17:10:25.343 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:25.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 326417514499 -lt 326417514499 2020-09-08T17:10:25.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:25.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-283467841543 2020-09-08T17:10:25.344 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:25.344 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:25.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-283467841543 2020-09-08T17:10:25.345 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:25.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=283467841543 2020-09-08T17:10:25.345 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 283467841543' 2020-09-08T17:10:25.346 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 283467841543 2020-09-08T17:10:25.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:25.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 283467841543 -lt 283467841543 2020-09-08T17:10:25.346 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:25.346 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149721 2020-09-08T17:10:25.347 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:25.347 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:25.550 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149721 2020-09-08T17:10:32.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:32.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149721 2020-09-08T17:10:32.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149721' 2020-09-08T17:10:32.087 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149721 2020-09-08T17:10:32.087 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:32.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149721 -lt 111669149721 2020-09-08T17:10:32.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:32.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:32.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:32.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:32.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:32.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:32.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:32.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:32.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:32.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:32.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:32.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:32.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:32.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:32.091 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:32.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:32.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:32.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:32.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: dd if=/dev/urandom bs=256 count=1 2020-09-08T17:10:32.092 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:10:32.092 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:10:32.092 INFO:tasks.workunit.client.0.smithi120.stderr:256 bytes copied, 3.2321e-05 s, 7.9 MB/s 2020-09-08T17:10:32.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:32.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:32.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:32.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:10:32.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:32.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:32.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:32.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:32.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:10:32.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:32.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:10:32.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:32.095 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:32.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:32.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:32.095 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:32.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:32.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:32.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:32.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:32.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:10:32.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:32.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:32.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:32.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:10:32.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:32.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:32.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:10:32.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:32.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:10:32.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:32.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:32.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:32.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:32.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:10:32.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:10:32.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:32.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:32.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:32.303 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:32.303 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:32.303 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:32.303 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:32.304 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:32.304 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:32.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:32.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:32.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:32.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:32.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:32.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:32.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:32.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:32.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:32.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:32.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:10:32.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:10:32.307 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:10:32.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:32.510 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:10:32.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:10:32.510 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:32.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:32.511 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:32.511 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:18.109+0000 7f6489e1eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:32.511 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:18.112+0000 7f6489e1eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:32.511 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:18.114+0000 7f6489e1eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:32.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:10:32.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:32.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:10:32.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:32.512 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:32.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:32.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:32.513 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:10:32.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:32.513 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:32.514 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:18.631+0000 7f6489e1eec0 -1 Falling back to public interface 2020-09-08T17:10:32.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:32.514 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:19.631+0000 7f6489e1eec0 -1 osd.1 78 log_to_monitors {default=true} 2020-09-08T17:10:32.514 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:32.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:32.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:32.515 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:32.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:32.515 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:10:32.516 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [76,78) [v2:127.0.0.1:6810/164711,v1:127.0.0.1:6811/164711] [v2:127.0.0.1:6812/164711,v1:127.0.0.1:6813/164711] exists,up 560e2dea-86b2-4218-b715-99f7a291333f 2020-09-08T17:10:32.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:32.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:32.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:32.516 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:10:32.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:32.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:32.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:32.517 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:32.517 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:32.518 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:32.518 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:32.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:32.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:32.518 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:32.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:32.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:32.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:32.722 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:32.722 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:32.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:32.723 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:32.723 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:32.723 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:32.723 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:32.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:32.724 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:32.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803809 2020-09-08T17:10:32.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803809 2020-09-08T17:10:32.724 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809' 2020-09-08T17:10:32.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:32.725 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:32.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318275 2020-09-08T17:10:32.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318275 2020-09-08T17:10:32.725 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809 1-352187318275' 2020-09-08T17:10:32.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:32.726 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:32.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=283467841547 2020-09-08T17:10:32.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 283467841547 2020-09-08T17:10:32.726 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809 1-352187318275 2-283467841547' 2020-09-08T17:10:32.727 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:32.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:32.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149725 2020-09-08T17:10:32.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149725 2020-09-08T17:10:32.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803809 1-352187318275 2-283467841547 3-111669149725' 2020-09-08T17:10:32.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:32.931 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803809 2020-09-08T17:10:32.931 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:32.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:32.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803809 2020-09-08T17:10:32.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:32.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803809 2020-09-08T17:10:32.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2020-09-08T17:10:32.932 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803809 2020-09-08T17:10:32.933 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:32.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803807 -lt 25769803809 2020-09-08T17:10:32.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:10:32.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:10:32.934 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:32.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803809 -lt 25769803809 2020-09-08T17:10:32.934 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:32.934 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-352187318275 2020-09-08T17:10:32.934 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:32.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:32.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-352187318275 2020-09-08T17:10:32.935 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:32.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318275 2020-09-08T17:10:32.935 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 352187318275' 2020-09-08T17:10:32.936 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 352187318275 2020-09-08T17:10:32.936 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:32.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318275 -lt 352187318275 2020-09-08T17:10:32.936 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:32.936 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-283467841547 2020-09-08T17:10:32.937 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:32.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:32.937 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-283467841547 2020-09-08T17:10:32.937 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:32.937 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=283467841547 2020-09-08T17:10:32.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 283467841547' 2020-09-08T17:10:32.938 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 283467841547 2020-09-08T17:10:32.938 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:32.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 283467841547 -lt 283467841547 2020-09-08T17:10:32.938 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:32.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149725 2020-09-08T17:10:32.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:32.939 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:32.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149725 2020-09-08T17:10:32.939 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:33.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149725 2020-09-08T17:10:33.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149725' 2020-09-08T17:10:33.142 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149725 2020-09-08T17:10:33.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:33.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149725 -lt 111669149725 2020-09-08T17:10:33.158 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:33.159 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:33.159 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:33.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:33.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:33.159 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:33.160 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:33.160 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:33.160 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:33.160 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:33.160 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:33.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T17:10:33.161 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:33.161 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:33.161 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:33.161 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T17:10:33.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' -1 2020-09-08T17:10:33.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T17:10:33.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=4 2020-09-08T17:10:33.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:10:33.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T17:10:33.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:10:33.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:33.367 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:33.367 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:33.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:33.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:33.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:33.368 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:33.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2020-09-08T17:10:33.369 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:33.369 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:33.369 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:33.369 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 5 2020-09-08T17:10:33.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 4 '!=' 4 2020-09-08T17:10:33.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2020-09-08T17:10:33.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2020-09-08T17:10:33.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-09-08T17:10:33.370 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-09-08T17:10:33.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-09-08T17:10:33.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2020-09-08T17:10:33.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-09-08T17:10:33.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2020-09-08T17:10:33.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2020-09-08T17:10:33.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2020-09-08T17:10:33.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:10:33.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2020-09-08T17:10:33.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:10:33.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:33.580 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:33.580 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:33.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:33.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:33.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:33.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:33.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:33.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:33.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:33.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:33.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:33.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:33.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:33.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:10:33.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:10:33.787 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:10:33.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:237: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-1-256 2020-09-08T17:10:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:10:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:10:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-1-256 2020-09-08T17:10:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:10:33.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:10:33.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-size-59146-1-256 td/test-erasure-eio/COPY 2020-09-08T17:10:33.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:10:33.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:10:33.789 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: expr 1 + 1 2020-09-08T17:10:33.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:240: rados_get_data_bad_size: shard_id=2 2020-09-08T17:10:33.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: set_size obj-size-59146-1-256 td/test-erasure-eio 2 256 add 2020-09-08T17:10:33.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:191: set_size: local objname=obj-size-59146-1-256 2020-09-08T17:10:33.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: shift 2020-09-08T17:10:33.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: local dir=td/test-erasure-eio 2020-09-08T17:10:33.994 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: shift 2020-09-08T17:10:35.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: local shard_id=2 2020-09-08T17:10:35.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: shift 2020-09-08T17:10:35.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: local bytes=256 2020-09-08T17:10:35.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: shift 2020-09-08T17:10:35.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: local mode=add 2020-09-08T17:10:35.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:201: set_size: local poolname=pool-jerasure 2020-09-08T17:10:35.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:10:35.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: get_osds pool-jerasure obj-size-59146-1-256 2020-09-08T17:10:35.220 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:10:35.220 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-size-59146-1-256 2020-09-08T17:10:35.220 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-size-59146-1-256 2020-09-08T17:10:35.220 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:10:35.221 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:10:35.221 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:35.221 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:10:35.221 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:10:35.221 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local -a initial_osds 2020-09-08T17:10:35.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local osd_id=2 2020-09-08T17:10:35.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: ceph osd set noout 2020-09-08T17:10:35.222 INFO:tasks.workunit.client.0.smithi120.stderr:noout is set 2020-09-08T17:10:35.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: '[' add = add ']' 2020-09-08T17:10:35.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:207: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:35.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:35.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:35.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=2 2020-09-08T17:10:35.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:35.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:35.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:35.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:35.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2020-09-08T17:10:35.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:35.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2020-09-08T17:10:35.428 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:35.428 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:35.428 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:35.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:35.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:35.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:35.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:35.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:35.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:35.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2020-09-08T17:10:35.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:35.430 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:35.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-59146-1-256 get-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:36.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2020-09-08T17:10:36.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:36.874 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-09-08T17:10:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:36.875 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:36.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:36.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:36.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:10:36.876 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:10:36.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:36.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:36.877 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:36.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:36.877 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:36.878 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:36.878 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:36.878 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:37.082 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:37.082 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:37.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:37.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:37.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:37.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:37.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:37.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:37.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:37.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:37.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:37.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:10:37.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-09-08T17:10:37.085 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.2 2020-09-08T17:10:37.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:37.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/2/whoami 2020-09-08T17:10:37.085 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-09-08T17:10:37.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:37.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:37.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:37.494 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:32.085+0000 7fe460943ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:37.494 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:32.087+0000 7fe460943ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:37.494 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:32.089+0000 7fe460943ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:37.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-09-08T17:10:37.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:10:37.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:10:37.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:10:37.495 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:10:37.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:37.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:10:37.496 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:10:37.496 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:37.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:10:37.497 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:32.609+0000 7fe460943ec0 -1 Falling back to public interface 2020-09-08T17:10:37.497 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:10:37.503 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:33.625+0000 7fe460943ec0 -1 osd.2 87 log_to_monitors {default=true} 2020-09-08T17:10:37.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:10:37.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:10:37.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:10:37.504 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:37.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:10:37.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:10:37.504 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 up in weight 1 up_from 92 up_thru 79 down_at 89 last_clean_interval [66,87) [v2:127.0.0.1:6818/166064,v1:127.0.0.1:6819/166064] [v2:127.0.0.1:6820/166064,v1:127.0.0.1:6821/166064] exists,up b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:10:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:10:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:10:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:10:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:10:37.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:10:37.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:10:37.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:10:37.506 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:10:37.506 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:10:37.506 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:10:37.507 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:10:37.507 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:10:37.507 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:10:37.507 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:10:37.507 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:10:37.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:10:37.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:10:37.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:10:37.508 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:10:37.508 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:10:37.509 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:10:37.509 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:10:37.509 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:10:37.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:10:37.509 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:37.714 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:10:37.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803814 2020-09-08T17:10:37.714 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803814 2020-09-08T17:10:37.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803814' 2020-09-08T17:10:37.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:37.715 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:10:37.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318280 2020-09-08T17:10:37.715 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318280 2020-09-08T17:10:37.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803814 1-352187318280' 2020-09-08T17:10:37.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:37.716 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:10:37.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=395136991235 2020-09-08T17:10:37.716 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 395136991235 2020-09-08T17:10:37.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803814 1-352187318280 2-395136991235' 2020-09-08T17:10:37.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:10:37.717 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:10:37.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149730 2020-09-08T17:10:37.717 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149730 2020-09-08T17:10:37.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803814 1-352187318280 2-395136991235 3-111669149730' 2020-09-08T17:10:37.718 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:37.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803814 2020-09-08T17:10:37.718 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:37.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:10:37.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803814 2020-09-08T17:10:37.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:37.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803814 2020-09-08T17:10:37.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2020-09-08T17:10:37.923 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803814 2020-09-08T17:10:37.923 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:37.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803811 -lt 25769803814 2020-09-08T17:10:37.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:10:37.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:10:37.924 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:10:37.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803814 -lt 25769803814 2020-09-08T17:10:37.924 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:37.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-352187318280 2020-09-08T17:10:37.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:37.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:10:37.925 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-352187318280 2020-09-08T17:10:37.926 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:37.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318280 2020-09-08T17:10:37.926 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 352187318280' 2020-09-08T17:10:37.926 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 352187318280 2020-09-08T17:10:37.927 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:10:37.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318280 -lt 352187318280 2020-09-08T17:10:37.929 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:37.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-395136991235 2020-09-08T17:10:37.929 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:37.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:10:37.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-395136991235 2020-09-08T17:10:37.930 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:37.930 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=395136991235 2020-09-08T17:10:37.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 395136991235' 2020-09-08T17:10:37.931 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 395136991235 2020-09-08T17:10:37.931 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:10:37.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 395136991235 -lt 395136991235 2020-09-08T17:10:37.931 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:10:37.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149730 2020-09-08T17:10:37.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:10:37.932 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:10:37.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149730 2020-09-08T17:10:37.932 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:10:37.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149730 2020-09-08T17:10:37.933 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149730' 2020-09-08T17:10:37.933 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149730 2020-09-08T17:10:37.933 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:10:38.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149730 -lt 111669149730 2020-09-08T17:10:38.278 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:10:38.279 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:38.279 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:38.759 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:10:38.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:10:38.777 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:10:38.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:10:38.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:10:38.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:10:38.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:10:38.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:10:39.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:10:39.134 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:10:39.134 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:10:39.135 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:10:39.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:10:39.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:10:39.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:10:39.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: dd if=/dev/urandom bs=256 count=1 2020-09-08T17:10:39.625 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records in 2020-09-08T17:10:39.626 INFO:tasks.workunit.client.0.smithi120.stderr:1+0 records out 2020-09-08T17:10:39.626 INFO:tasks.workunit.client.0.smithi120.stderr:256 bytes copied, 3.0979e-05 s, 8.3 MB/s 2020-09-08T17:10:39.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:215: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:39.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:10:39.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:10:39.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=2 2020-09-08T17:10:39.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:10:39.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:39.627 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:10:39.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:10:39.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2020-09-08T17:10:39.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:10:39.628 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2020-09-08T17:10:39.628 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:10:39.629 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:10:39.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:10:39.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:10:39.629 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:10:39.729 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:10:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:10:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:10:39.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2020-09-08T17:10:39.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:10:39.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:39.731 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-59146-1-256 set-bytes td/test-erasure-eio/CORRUPT 2020-09-08T17:10:41.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2020-09-08T17:10:41.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:10:41.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:10:41.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-09-08T17:10:41.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:10:41.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:10:41.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:10:41.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:10:41.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:10:41.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:10:41.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:10:41.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:10:41.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:10:41.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:10:41.614 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:10:41.614 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:10:41.614 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:10:41.615 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:10:41.615 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:10:41.615 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:10:41.615 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:10:41.615 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:41.818 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:10:42.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:10:42.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:10:42.022 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:10:42.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:10:42.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:10:42.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:42.023 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:10:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:10:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:10:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-09-08T17:10:42.024 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.2 2020-09-08T17:10:42.025 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:10:42.025 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/2/whoami 2020-09-08T17:10:42.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-09-08T17:10:42.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:10:42.027 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:10:42.028 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:10:42.028 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:41.638+0000 7f76bf94fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:10:42.230 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:41.639+0000 7f76bf94fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:11:30.018 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:41.640+0000 7f76bf94fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:11:30.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-09-08T17:11:30.222 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:11:30.227 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:11:30.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:11:30.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:11:30.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:11:30.228 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:11:30.228 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:11:30.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:11:30.229 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:11:30.283 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:42.159+0000 7f76bf94fec0 -1 Falling back to public interface 2020-09-08T17:11:30.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:11:30.283 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:43.012+0000 7f76bf94fec0 -1 osd.2 94 log_to_monitors {default=true} 2020-09-08T17:11:30.283 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:11:30.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:11:30.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:11:30.284 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:11:30.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:11:30.284 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:11:30.285 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 up in weight 1 up_from 98 up_thru 79 down_at 95 last_clean_interval [92,94) [v2:127.0.0.1:6818/167195,v1:127.0.0.1:6819/167195] [v2:127.0.0.1:6820/167195,v1:127.0.0.1:6821/167195] exists,up b6975801-1d30-4729-a6a6-69d8802ecfb7 2020-09-08T17:11:30.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:11:30.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:11:30.285 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:11:30.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:11:30.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:11:30.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:11:30.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:11:30.286 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:11:30.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:11:30.287 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:11:30.287 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:11:30.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:11:30.287 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:11:30.288 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:11:30.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:11:30.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:11:30.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:11:30.288 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:11:30.289 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:11:30.289 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:11:30.289 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:11:30.289 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:11:30.289 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:11:30.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:11:30.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:11:30.290 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:11:30.290 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803817 2020-09-08T17:11:30.291 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803817 2020-09-08T17:11:30.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803817' 2020-09-08T17:11:30.292 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:11:30.292 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:11:30.293 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318283 2020-09-08T17:11:30.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318283 2020-09-08T17:11:30.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803817 1-352187318283' 2020-09-08T17:11:30.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:11:30.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:11:30.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=420906795011 2020-09-08T17:11:30.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 420906795011 2020-09-08T17:11:30.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803817 1-352187318283 2-420906795011' 2020-09-08T17:11:30.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:11:30.296 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:11:30.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149734 2020-09-08T17:11:30.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149734 2020-09-08T17:11:30.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803817 1-352187318283 2-420906795011 3-111669149734' 2020-09-08T17:11:30.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:11:30.298 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803817 2020-09-08T17:11:30.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:11:30.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:11:30.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803817 2020-09-08T17:11:30.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:11:30.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803817 2020-09-08T17:11:30.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2020-09-08T17:11:30.301 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803817 2020-09-08T17:11:30.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:11:30.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803817 -lt 25769803817 2020-09-08T17:11:30.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:11:30.301 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-352187318283 2020-09-08T17:11:30.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:11:30.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:11:30.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-352187318283 2020-09-08T17:11:30.302 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:11:30.302 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318283 2020-09-08T17:11:30.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 352187318283' 2020-09-08T17:11:30.303 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 352187318283 2020-09-08T17:11:30.303 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:11:30.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318284 -lt 352187318283 2020-09-08T17:11:30.303 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:11:30.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-420906795011 2020-09-08T17:11:30.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:11:30.304 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:11:30.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-420906795011 2020-09-08T17:11:30.304 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:11:30.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=420906795011 2020-09-08T17:11:30.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 420906795011' 2020-09-08T17:11:30.305 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 420906795011 2020-09-08T17:11:30.305 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:11:30.305 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 420906795011 -lt 420906795011 2020-09-08T17:11:30.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:11:30.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149734 2020-09-08T17:11:30.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:11:30.306 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:11:30.306 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149734 2020-09-08T17:11:30.307 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:11:30.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149734 2020-09-08T17:11:30.307 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149734' 2020-09-08T17:11:30.307 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149734 2020-09-08T17:11:30.307 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:11:30.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149732 -lt 111669149734 2020-09-08T17:11:30.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:11:30.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:11:30.308 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:11:30.308 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149734 -lt 111669149734 2020-09-08T17:11:30.309 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:11:30.309 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:11:30.309 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:11:30.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:11:30.309 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:11:30.310 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:11:30.310 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:11:30.310 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:11:30.310 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:11:30.310 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:11:30.311 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:11:30.311 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:11:30.311 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:11:30.312 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:11:30.312 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:11:30.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:11:30.312 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:11:30.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:11:30.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: rm -f td/test-erasure-eio/CORRUPT 2020-09-08T17:11:30.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: ceph osd unset noout 2020-09-08T17:11:30.313 INFO:tasks.workunit.client.0.smithi120.stderr:noout is unset 2020-09-08T17:11:30.313 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-59146-1-256 fail 2020-09-08T17:11:30.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:11:30.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:11:30.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-size-59146-1-256 2020-09-08T17:11:30.314 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:11:30.518 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:11:30.519 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-size-59146-1-256 td/test-erasure-eio/COPY 2020-09-08T17:11:30.519 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-size-59146-1-256: (5) Input/output error 2020-09-08T17:11:30.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:11:30.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:11:30.520 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:11:30.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:11:30.521 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:11:30.521 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:11:30.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:11:30.522 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:11:30.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:11:30.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:11:30.522 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:11:30.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:11:30.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:11:30.523 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:11:30.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:11:30.523 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:11:30.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:11:30.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:11:30.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:11:30.524 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:11:30.524 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:11:30.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:11:30.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:11:30.525 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:11:30.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:11:30.525 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:11:30.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:11:30.526 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:11:30.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:11:30.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:11:30.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:11:30.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:11:30.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:30.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:30.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:11:30.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:11:30.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:11:30.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:11:30.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:11:30.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:11:30.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:11:30.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:11:30.730 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:11:48.526 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:11:48.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:11:48.527 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:11:48.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:11:48.527 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:11:48.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:11:48.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:11:48.528 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:11:48.528 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:11:48.529 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:11:48.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:11:48.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:11:48.529 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:11:48.529 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:11:48.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:11:48.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:11:48.530 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:11:48.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:11:48.530 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:11:48.531 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:11:48.531 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:11:48.531 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:48.734 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:48.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:11:48.734 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:11:48.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:11:48.735 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:11:48.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:11:48.735 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:48.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:48.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:11:48.736 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:11:48.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:11:48.736 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:11:48.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:11:48.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:11:48.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:11:48.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:11:48.737 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:11:48.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:11:48.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:11:48.738 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:11:48.738 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:11:48.738 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:11:48.739 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:11:48.942 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:11:48.942 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:48.942 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:48.943 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:11:48.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:11:48.943 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:11:48.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:11:48.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:11:48.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:11:48.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:11:48.944 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:11:48.945 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:11:48.945 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:11:48.945 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:11:48.945 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:11:48.945 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:48.946 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:48.946 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:11:48.946 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:11:48.946 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:11:49.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:11:49.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:11:49.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:11:49.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:11:49.151 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:11:49.151 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:11:49.151 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:11:49.151 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:11:49.152 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:11:49.152 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:49.152 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:49.152 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:11:49.152 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:11:49.153 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:11:49.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:11:49.153 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:11:49.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:11:49.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:11:49.154 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:11:49.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:11:49.155 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:11:49.155 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:11:49.155 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:11:49.156 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:11:49.156 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:11:49.156 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:49.156 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:49.156 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:11:49.157 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:11:49.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:11:49.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:11:49.157 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:11:49.157 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:11:49.158 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:11:49.158 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:11:49.158 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:11:49.158 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:11:49.158 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:11:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:11:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:11:49.159 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:11:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:11:49.159 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:11:49.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:12:36.500 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2020-09-08T17:12:36.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:254: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2020-09-08T17:12:36.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:255: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2020-09-08T17:12:36.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=4 2020-09-08T17:12:36.501 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:12:36.502 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 4 - 1 2020-09-08T17:12:36.502 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 3 2020-09-08T17:12:36.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:12:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:12:36.503 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:36.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:12:36.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:36.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:36.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:36.504 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:36.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:12:36.505 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:12:36.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:36.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:36.706 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:36.707 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:36.707 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:36.707 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:36.707 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:36.707 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:36.708 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:36.708 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:36.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:36.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:36.708 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:36.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:36.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:36.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:36.709 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:36.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:36.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:36.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:36.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:36.710 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:12:36.914 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:36.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5b374954-7a0d-4a41-a495-cf94fbfea312 2020-09-08T17:12:36.914 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 5b374954-7a0d-4a41-a495-cf94fbfea312' 2020-09-08T17:12:36.914 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:36.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAdu1dfRdHyJhAA39NOjcO9Y+gzuSM/tIdtig== 2020-09-08T17:12:36.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAdu1dfRdHyJhAA39NOjcO9Y+gzuSM/tIdtig=="}' 2020-09-08T17:12:36.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5b374954-7a0d-4a41-a495-cf94fbfea312 -i td/test-erasure-eio/0/new.json 2020-09-08T17:12:36.915 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:12:36.916 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAdu1dfRdHyJhAA39NOjcO9Y+gzuSM/tIdtig== --osd-uuid 5b374954-7a0d-4a41-a495-cf94fbfea312 2020-09-08T17:12:36.916 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:54.004+0000 7f9b73145ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:36.916 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:54.006+0000 7f9b73145ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:36.916 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:54.007+0000 7f9b73145ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:36.916 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:54.008+0000 7f9b73145ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:12:36.917 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:54.008+0000 7f9b73145ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:12:36.917 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:54.008+0000 7f9b73145ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:12:36.917 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:54.115+0000 7f9b73145ec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:12:36.917 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:12:37.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:37.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:12:37.118 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:37.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:12:37.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:37.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:37.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:37.119 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:37.120 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:58.527+0000 7f3be8a6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.120 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:58.529+0000 7f3be8a6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.120 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:58.530+0000 7f3be8a6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:12:37.120 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:37.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:12:37.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:37.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:37.121 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:37.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:12:37.122 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:59.046+0000 7f3be8a6bec0 -1 Falling back to public interface 2020-09-08T17:12:37.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:37.123 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:10:59.814+0000 7f3be8a6bec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:12:37.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:37.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:37.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:12:37.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:37.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:37.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:37.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:37.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:12:37.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:37.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:37.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:12:37.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:37.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:12:37.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:37.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:37.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:37.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:37.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:12:37.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:12:37.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:37.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:37.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:37.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:37.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:37.539 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:37.539 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:37.540 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:37.540 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:37.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:37.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:37.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:37.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:37.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:37.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:37.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:37.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:37.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:37.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:37.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:37.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:37.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:12:37.543 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:37.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f8a26079-28c5-4017-a451-274f9fc83418 2020-09-08T17:12:37.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 f8a26079-28c5-4017-a451-274f9fc83418' 2020-09-08T17:12:37.543 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:37.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAku1dfWKwPKBAAbRemh6d1MfkJ7OzZZ1uD3A== 2020-09-08T17:12:37.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAku1dfWKwPKBAAbRemh6d1MfkJ7OzZZ1uD3A=="}' 2020-09-08T17:12:37.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f8a26079-28c5-4017-a451-274f9fc83418 -i td/test-erasure-eio/1/new.json 2020-09-08T17:12:37.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:12:37.544 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAku1dfWKwPKBAAbRemh6d1MfkJ7OzZZ1uD3A== --osd-uuid f8a26079-28c5-4017-a451-274f9fc83418 2020-09-08T17:12:37.545 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:01.222+0000 7fcc5993bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.545 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:01.224+0000 7fcc5993bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.545 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:01.225+0000 7fcc5993bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.545 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:01.226+0000 7fcc5993bec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:12:37.546 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:01.226+0000 7fcc5993bec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:12:37.546 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:01.226+0000 7fcc5993bec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:12:37.546 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:01.275+0000 7fcc5993bec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:12:37.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:12:37.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:37.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:12:37.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:37.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:12:37.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:37.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:37.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:37.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:37.548 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:05.801+0000 7fddf0b6cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.548 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:05.803+0000 7fddf0b6cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.548 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:05.804+0000 7fddf0b6cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:12:37.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:37.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:12:37.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:37.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:37.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:37.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:12:37.550 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:06.319+0000 7fddf0b6cec0 -1 Falling back to public interface 2020-09-08T17:12:37.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:37.551 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:07.107+0000 7fddf0b6cec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:12:37.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:37.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:37.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:12:37.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:37.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:37.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:37.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:37.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:12:37.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:37.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:37.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:12:37.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:37.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:12:37.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:37.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:37.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:37.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:37.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:12:37.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:12:37.555 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:37.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:37.556 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:37.556 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:37.556 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:37.556 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:37.557 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:37.557 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:37.557 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:37.557 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:37.557 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:37.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:37.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:37.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:37.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:37.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:37.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:37.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:37.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:37.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:37.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:37.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:12:37.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:37.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b1fde163-f23b-47a3-9d6e-6c90e1bb08c0 2020-09-08T17:12:37.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 b1fde163-f23b-47a3-9d6e-6c90e1bb08c0' 2020-09-08T17:12:37.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:37.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAru1dfnQYcORAACSe/iJHI8UdEzYLkjIa4PQ== 2020-09-08T17:12:37.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAru1dfnQYcORAACSe/iJHI8UdEzYLkjIa4PQ=="}' 2020-09-08T17:12:37.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b1fde163-f23b-47a3-9d6e-6c90e1bb08c0 -i td/test-erasure-eio/2/new.json 2020-09-08T17:12:37.561 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:12:37.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAru1dfnQYcORAACSe/iJHI8UdEzYLkjIa4PQ== --osd-uuid b1fde163-f23b-47a3-9d6e-6c90e1bb08c0 2020-09-08T17:12:37.562 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:08.722+0000 7fa137105ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.562 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:08.723+0000 7fa137105ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.562 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:08.724+0000 7fa137105ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.563 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:08.725+0000 7fa137105ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:12:37.563 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:08.725+0000 7fa137105ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:12:37.766 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:08.725+0000 7fa137105ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:12:37.766 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:08.768+0000 7fa137105ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:12:37.766 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:12:37.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:37.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:12:37.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:37.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:12:37.767 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:37.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:37.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:37.768 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:37.768 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:13.544+0000 7f6a8b264ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.768 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:13.545+0000 7f6a8b264ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.769 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:13.546+0000 7f6a8b264ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:12:37.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:37.769 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:12:37.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:37.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:37.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:37.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:12:37.771 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:14.065+0000 7f6a8b264ec0 -1 Falling back to public interface 2020-09-08T17:12:37.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:37.771 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:14.810+0000 7f6a8b264ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:12:37.771 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:37.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:37.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:12:37.772 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:37.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:37.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:37.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:37.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:12:37.773 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:37.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:37.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:12:37.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:37.774 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:12:37.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:37.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:37.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:37.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:37.775 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:12:37.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:12:37.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:37.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:37.776 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:37.776 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:37.777 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:37.777 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:37.777 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:37.777 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:37.777 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:37.778 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:37.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:37.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:37.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:37.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:37.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:37.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:37.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:37.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:37.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:37.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:37.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:37.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:12:37.780 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:37.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=970a623e-976e-4a77-b14d-fb018a45f1af 2020-09-08T17:12:37.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 970a623e-976e-4a77-b14d-fb018a45f1af' 2020-09-08T17:12:37.781 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:37.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAzu1df7OBFKRAAmzDTBbk0i+7PU4b8xLplSQ== 2020-09-08T17:12:37.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAzu1df7OBFKRAAmzDTBbk0i+7PU4b8xLplSQ=="}' 2020-09-08T17:12:37.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 970a623e-976e-4a77-b14d-fb018a45f1af -i td/test-erasure-eio/3/new.json 2020-09-08T17:12:37.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:12:37.782 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAzu1df7OBFKRAAmzDTBbk0i+7PU4b8xLplSQ== --osd-uuid 970a623e-976e-4a77-b14d-fb018a45f1af 2020-09-08T17:12:37.782 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:16.555+0000 7f3f25daeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.782 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:16.588+0000 7f3f25daeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.782 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:16.589+0000 7f3f25daeec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:16.590+0000 7f3f25daeec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:12:37.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:16.590+0000 7f3f25daeec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:12:37.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:16.590+0000 7f3f25daeec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:12:37.783 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:16.722+0000 7f3f25daeec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:12:37.783 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:12:37.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:37.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:12:37.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:37.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:12:37.784 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:37.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:37.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:37.785 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:37.785 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:21.946+0000 7fc331820ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.785 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:21.948+0000 7fc331820ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.786 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:21.950+0000 7fc331820ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:37.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:12:37.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:37.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:12:37.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:37.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:37.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:37.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.787 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:12:37.788 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:22.467+0000 7fc331820ec0 -1 Falling back to public interface 2020-09-08T17:12:37.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:37.788 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:23.235+0000 7fc331820ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:12:37.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:37.788 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:37.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:12:37.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:37.789 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:37.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:37.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2020-09-08T17:12:37.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:37.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:12:37.790 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:37.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:37.791 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:37.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:12:37.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:12:37.791 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:12:37.792 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:37.792 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:37.792 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:37.792 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:12:37.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:12:37.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:12:37.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:12:37.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:257: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2020-09-08T17:12:37.793 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2020-09-08T17:12:37.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:12:37.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:12:37.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=2 2020-09-08T17:12:37.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:12:37.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=1 2020-09-08T17:12:37.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:12:37.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2020-09-08T17:12:37.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:12:37.795 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:12:37.795 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:12:38.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:12:38.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:12:38.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:12:38.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:12:38.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:12:38.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:12:38.003 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:12:38.004 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:12:38.004 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:12:38.004 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:12:38.004 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:12:38.004 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:12:38.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:12:38.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:12:38.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:12:38.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:12:38.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:12:38.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:12:38.007 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:12:38.007 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:12:38.007 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:12:38.007 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:12:38.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:38.008 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:12:38.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803785 2020-09-08T17:12:38.008 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803785 2020-09-08T17:12:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785' 2020-09-08T17:12:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:38.009 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:12:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607559 2020-09-08T17:12:38.009 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607559 2020-09-08T17:12:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607559' 2020-09-08T17:12:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:38.010 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:12:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411333 2020-09-08T17:12:38.010 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411333 2020-09-08T17:12:38.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607559 2-77309411333' 2020-09-08T17:12:38.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:38.011 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:12:38.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149699 2020-09-08T17:12:38.011 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149699 2020-09-08T17:12:38.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607559 2-77309411333 3-111669149699' 2020-09-08T17:12:38.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:12:38.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803785 2020-09-08T17:12:38.012 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:12:38.012 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:12:38.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803785 2020-09-08T17:12:38.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:12:38.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803785 2020-09-08T17:12:38.013 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2020-09-08T17:12:38.013 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:12:38.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803783 -lt 25769803785 2020-09-08T17:12:38.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:12:38.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:12:38.014 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:12:38.015 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:12:38.015 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:10:51.391+0000 7f9e3eb226c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:12:38.015 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 5b374954-7a0d-4a41-a495-cf94fbfea312 2020-09-08T17:12:38.016 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:12:38.016 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:12:38.016 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:12:38.016 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:12:38.016 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:12:38.017 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/169045,v1:127.0.0.1:6803/169045] [v2:127.0.0.1:6804/169045,v1:127.0.0.1:6805/169045] exists,up 5b374954-7a0d-4a41-a495-cf94fbfea312 2020-09-08T17:12:38.017 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 f8a26079-28c5-4017-a451-274f9fc83418 2020-09-08T17:12:38.017 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:12:38.017 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:12:38.017 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:12:38.018 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:12:38.018 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:12:38.018 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/170039,v1:127.0.0.1:6811/170039] [v2:127.0.0.1:6812/170039,v1:127.0.0.1:6813/170039] exists,up f8a26079-28c5-4017-a451-274f9fc83418 2020-09-08T17:12:38.018 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 b1fde163-f23b-47a3-9d6e-6c90e1bb08c0 2020-09-08T17:12:38.018 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:12:38.019 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:12:38.019 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:12:38.019 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:12:38.019 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:12:38.019 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/171033,v1:127.0.0.1:6819/171033] [v2:127.0.0.1:6820/171033,v1:127.0.0.1:6821/171033] exists,up b1fde163-f23b-47a3-9d6e-6c90e1bb08c0 2020-09-08T17:12:38.020 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 970a623e-976e-4a77-b14d-fb018a45f1af 2020-09-08T17:12:38.020 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:12:38.020 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:12:38.020 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:12:38.020 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:12:38.021 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:12:38.021 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:12:38.021 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/172027,v1:127.0.0.1:6827/172027] [v2:127.0.0.1:6828/172027,v1:127.0.0.1:6829/172027] exists,up 970a623e-976e-4a77-b14d-fb018a45f1af 2020-09-08T17:12:38.021 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:12:38.021 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:10:59.052+0000 7f3be8a6bec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:12:38.022 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803785 2020-09-08T17:12:38.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803785 -lt 25769803785 2020-09-08T17:12:38.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:12:38.125 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607559 2020-09-08T17:12:38.125 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:12:38.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:12:38.126 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607559 2020-09-08T17:12:38.126 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:12:38.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607559 2020-09-08T17:12:38.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607559' 2020-09-08T17:12:38.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:12:38.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607559 -lt 51539607559 2020-09-08T17:12:38.127 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:12:38.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411333 2020-09-08T17:12:38.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:12:38.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:12:38.128 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411333 2020-09-08T17:12:38.128 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:12:38.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411333 2020-09-08T17:12:38.128 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411333' 2020-09-08T17:12:38.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:12:38.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411333 -lt 77309411333 2020-09-08T17:12:38.129 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:12:38.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149699 2020-09-08T17:12:38.129 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:12:38.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:12:38.130 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149699 2020-09-08T17:12:38.130 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:12:38.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149699 2020-09-08T17:12:38.130 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2020-09-08T17:12:38.131 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:12:38.131 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149700 -lt 111669149699 2020-09-08T17:12:38.131 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:12:38.131 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:12:38.131 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:12:38.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:12:38.132 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:12:38.132 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:12:38.132 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:12:38.132 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:12:38.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:12:38.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:12:38.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:12:38.133 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:12:38.133 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:12:38.134 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:12:38.134 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:12:38.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:12:38.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:12:38.134 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:12:38.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:260: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2020-09-08T17:12:38.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:261: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2020-09-08T17:12:38.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:146: rados_put_get_data: local inject=eio 2020-09-08T17:12:38.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: shift 2020-09-08T17:12:38.135 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: local dir=td/test-erasure-eio 2020-09-08T17:12:38.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: shift 2020-09-08T17:12:38.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: local shard_id=0 2020-09-08T17:12:38.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: shift 2020-09-08T17:12:38.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: local arg= 2020-09-08T17:12:38.136 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:156: rados_put_get_data: local poolname=pool-jerasure 2020-09-08T17:12:38.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local objname=obj-eio-59146-0 2020-09-08T17:12:38.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-59146-0 2020-09-08T17:12:38.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:12:38.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:12:38.137 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-eio-59146-0 2020-09-08T17:12:38.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:12:38.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:12:38.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:12:38.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:12:38.138 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:12:38.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:12:38.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:12:38.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:12:38.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-eio-59146-0 td/test-erasure-eio/ORIGINAL 2020-09-08T17:12:38.139 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-59146-0 td/test-erasure-eio 0 2020-09-08T17:12:38.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:12:38.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:12:38.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:12:38.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:12:38.140 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:12:38.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:12:38.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-59146-0 2020-09-08T17:12:38.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:12:38.141 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:12:38.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:12:38.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2020-09-08T17:12:38.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:12:38.142 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:12:38.142 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj-eio-59146-0 2020-09-08T17:12:38.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:12:38.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-59146-0 2020-09-08T17:12:38.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-eio-59146-0 2020-09-08T17:12:38.143 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:12:38.143 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:12:38.144 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:12:38.144 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:12:38.144 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:12:38.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:12:38.144 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2020-09-08T17:12:38.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:12:38.145 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/3/type 2020-09-08T17:12:38.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:12:38.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2020-09-08T17:12:38.145 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:12:38.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2020-09-08T17:12:38.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:12:38.146 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:12:38.146 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:12:38.146 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2020-09-08T17:12:38.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:12:38.147 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:12:38.147 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:38.147 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.147 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.148 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:12:38.148 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:12:38.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:12:38.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:12:38.148 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:12:38.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2020-09-08T17:12:38.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2020-09-08T17:12:38.149 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2020-09-08T17:12:38.149 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:38.149 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.150 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.150 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.3.asok 2020-09-08T17:12:38.150 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:12:38.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-59146-0 0 2020-09-08T17:12:38.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-59146-0 2020-09-08T17:12:38.354 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:12:38.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:12:38.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-eio-59146-0 2020-09-08T17:12:38.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:12:38.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:12:38.355 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-eio-59146-0 td/test-erasure-eio/COPY 2020-09-08T17:12:38.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:12:38.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:12:38.356 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:162: rados_put_get_data: '[' '' = recovery ']' 2020-09-08T17:12:38.356 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:180: rados_put_get_data: expr 0 + 1 2020-09-08T17:12:38.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:180: rados_put_get_data: shard_id=1 2020-09-08T17:12:38.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-59146-0 td/test-erasure-eio 1 2020-09-08T17:12:38.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:12:38.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:12:38.357 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:12:38.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:12:38.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:12:38.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:12:38.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-59146-0 2020-09-08T17:12:38.358 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:12:38.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:12:38.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:12:38.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2020-09-08T17:12:38.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:12:38.359 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:12:38.360 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj-eio-59146-0 2020-09-08T17:12:38.360 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:12:38.360 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-59146-0 2020-09-08T17:12:38.360 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-eio-59146-0 2020-09-08T17:12:38.360 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:12:38.361 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:12:38.361 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:12:38.361 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:12:38.361 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:12:38.361 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:12:38.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2020-09-08T17:12:38.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:12:38.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/1/type 2020-09-08T17:12:38.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:12:38.362 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-09-08T17:12:38.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:12:38.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2020-09-08T17:12:38.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:12:38.363 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:12:38.363 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:12:38.364 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2020-09-08T17:12:38.364 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T17:12:38.364 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T17:12:38.364 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:38.364 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.365 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.365 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T17:12:38.365 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:12:38.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:12:38.365 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:12:38.366 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:12:38.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2020-09-08T17:12:38.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T17:12:38.366 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T17:12:38.366 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:38.367 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.367 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.367 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T17:12:38.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:12:38.367 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-59146-0 1 2020-09-08T17:12:38.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-59146-0 fail 2020-09-08T17:12:38.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:12:38.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:12:38.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-eio-59146-0 2020-09-08T17:12:38.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:12:38.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:12:38.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-eio-59146-0 td/test-erasure-eio/COPY 2020-09-08T17:12:38.571 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-eio-59146-0: (5) Input/output error 2020-09-08T17:12:38.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:12:38.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:12:38.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:12:38.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:12:38.572 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:12:38.573 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:12:38.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:12:38.573 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:12:38.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:12:38.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:12:38.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:12:38.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:12:38.574 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:12:38.574 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:12:38.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:12:38.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:12:38.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:12:38.575 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:12:38.575 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:12:38.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:12:38.576 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:12:38.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:12:38.576 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:12:38.576 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:12:38.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:12:38.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:12:38.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:12:38.577 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:12:38.577 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:12:38.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:12:38.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:12:38.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:12:38.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.578 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:12:38.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:12:38.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:12:38.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:12:38.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:12:38.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:12:38.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:12:38.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:12:38.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:12:38.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:12:38.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:12:38.581 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:12:38.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:12:38.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:12:38.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:12:38.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:12:38.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:12:38.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:12:38.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:12:38.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:12:38.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:12:38.583 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:12:38.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:12:38.583 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:12:38.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:12:38.584 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:12:38.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:12:38.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:12:38.584 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:12:38.585 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:12:38.585 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.585 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:12:38.585 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:12:38.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:12:38.586 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:12:38.586 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:12:38.586 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.586 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:12:38.587 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:12:38.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:12:38.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:12:38.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:12:38.587 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:12:38.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:12:38.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:12:38.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:12:38.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:12:38.588 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:12:38.589 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:12:38.589 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:12:38.589 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:38.589 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:38.590 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:38.590 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.590 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.590 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.590 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:12:38.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:12:38.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:12:38.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:12:38.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:12:38.591 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:12:38.592 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:12:38.592 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:12:38.592 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:12:38.592 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:12:38.592 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:38.593 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.593 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.593 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:12:38.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:12:38.593 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:12:38.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:12:38.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:12:38.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:12:38.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:12:38.594 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:12:38.595 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:12:38.595 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:12:38.595 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:12:38.595 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:38.595 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.596 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.596 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:12:38.596 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:12:38.596 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:12:38.596 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:12:38.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:12:38.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:12:38.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:12:38.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:12:38.597 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:12:38.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:12:38.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:12:38.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:38.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:38.599 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:38.599 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.599 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:12:38.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:12:38.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:12:38.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:12:38.600 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:12:38.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:12:38.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:12:38.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:12:38.601 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:12:38.601 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:38.601 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.602 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.602 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:12:38.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:12:38.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:12:38.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:12:38.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2020-09-08T17:12:38.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:266: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2020-09-08T17:12:38.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:267: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2020-09-08T17:12:38.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=4 2020-09-08T17:12:38.603 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:12:38.604 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 4 - 1 2020-09-08T17:12:38.604 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 3 2020-09-08T17:12:38.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:38.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:12:38.604 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:38.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:38.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:12:38.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:38.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:12:38.605 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:38.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:38.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:38.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:38.606 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:12:38.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:12:38.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:38.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:38.607 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:38.607 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:38.608 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:38.608 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:38.608 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:38.608 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.608 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.609 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:38.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:38.609 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:38.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:38.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:38.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:38.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:38.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:38.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:38.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:12:38.611 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:38.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8e876070-781c-4073-ae25-749f1e81ebb9 2020-09-08T17:12:38.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 8e876070-781c-4073-ae25-749f1e81ebb9' 2020-09-08T17:12:38.612 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:38.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBKu1df/242GxAAGRup0r7U/bVE7mI2I5kMxA== 2020-09-08T17:12:38.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBKu1df/242GxAAGRup0r7U/bVE7mI2I5kMxA=="}' 2020-09-08T17:12:38.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8e876070-781c-4073-ae25-749f1e81ebb9 -i td/test-erasure-eio/0/new.json 2020-09-08T17:12:38.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:12:38.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBKu1df/242GxAAGRup0r7U/bVE7mI2I5kMxA== --osd-uuid 8e876070-781c-4073-ae25-749f1e81ebb9 2020-09-08T17:12:38.613 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:38.809+0000 7f7f836efec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.613 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:38.811+0000 7f7f836efec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.614 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:38.812+0000 7f7f836efec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.614 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:38.812+0000 7f7f836efec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:12:38.614 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:38.812+0000 7f7f836efec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:12:38.614 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:38.812+0000 7f7f836efec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:12:38.614 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:38.851+0000 7f7f836efec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:12:38.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:12:38.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:38.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:12:38.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:38.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:12:38.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:38.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:38.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:38.616 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:43.293+0000 7f607a3c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.617 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:43.295+0000 7f607a3c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.617 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:43.297+0000 7f607a3c5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:12:38.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:38.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:12:38.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:38.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:38.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:38.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:38.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:38.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:12:38.619 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:43.812+0000 7f607a3c5ec0 -1 Falling back to public interface 2020-09-08T17:12:38.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:38.619 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:44.569+0000 7f607a3c5ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:12:38.619 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:38.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:38.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:38.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:38.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:12:38.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:38.620 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:38.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:38.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:38.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:12:38.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:38.621 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:38.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:12:38.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:38.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:12:38.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:38.622 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:38.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:38.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:38.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:12:38.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:12:38.623 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:38.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:38.624 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:38.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:38.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:38.624 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:38.625 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:38.625 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.625 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.625 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.625 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:38.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:38.626 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:38.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:38.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:38.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:38.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:38.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:38.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:38.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:12:38.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:38.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3e0da9c2-1703-49be-868e-b0263a7c279e 2020-09-08T17:12:38.832 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 3e0da9c2-1703-49be-868e-b0263a7c279e' 2020-09-08T17:12:38.832 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:38.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBRu1df7VEnFxAAwOGsFO+Ay4SFgzN0W3cWSg== 2020-09-08T17:12:38.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBRu1df7VEnFxAAwOGsFO+Ay4SFgzN0W3cWSg=="}' 2020-09-08T17:12:38.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3e0da9c2-1703-49be-868e-b0263a7c279e -i td/test-erasure-eio/1/new.json 2020-09-08T17:12:38.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:12:38.833 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBRu1df7VEnFxAAwOGsFO+Ay4SFgzN0W3cWSg== --osd-uuid 3e0da9c2-1703-49be-868e-b0263a7c279e 2020-09-08T17:12:38.834 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:46.010+0000 7f7809521ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.834 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:46.029+0000 7f7809521ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.834 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:46.030+0000 7f7809521ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.834 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:46.031+0000 7f7809521ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:12:38.835 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:46.031+0000 7f7809521ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:12:38.835 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:46.031+0000 7f7809521ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:12:38.835 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:46.095+0000 7f7809521ec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:12:38.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:12:38.835 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:38.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:12:38.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:38.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:12:38.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:38.836 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:38.837 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:38.837 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:50.604+0000 7f99478ceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.837 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:50.606+0000 7f99478ceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.837 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:50.607+0000 7f99478ceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:12:38.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:38.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:12:38.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:38.838 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:38.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:38.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:38.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:38.839 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:12:38.839 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:51.123+0000 7f99478ceec0 -1 Falling back to public interface 2020-09-08T17:12:38.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:38.840 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:51.878+0000 7f99478ceec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:12:38.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:38.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:38.840 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:38.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:38.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:12:38.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:38.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:38.841 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:38.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:38.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:12:38.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:38.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:38.842 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:12:38.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:38.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:12:38.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:38.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:38.843 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:38.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:38.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:12:38.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:12:38.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:38.844 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:38.845 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:38.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:38.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:38.845 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:38.846 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:38.846 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.846 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.846 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.846 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:38.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:38.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:38.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:38.847 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:38.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:38.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:38.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:38.848 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:38.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:12:38.849 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:38.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7d5cf1c2-2f69-4487-8203-9e664fd11f01 2020-09-08T17:12:38.849 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 7d5cf1c2-2f69-4487-8203-9e664fd11f01' 2020-09-08T17:12:38.849 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:38.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBYu1df2b8JLBAAbIMMz3gmMW2S9bBGnKUnGg== 2020-09-08T17:12:38.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBYu1df2b8JLBAAbIMMz3gmMW2S9bBGnKUnGg=="}' 2020-09-08T17:12:38.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7d5cf1c2-2f69-4487-8203-9e664fd11f01 -i td/test-erasure-eio/2/new.json 2020-09-08T17:12:38.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:12:38.850 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBYu1df2b8JLBAAbIMMz3gmMW2S9bBGnKUnGg== --osd-uuid 7d5cf1c2-2f69-4487-8203-9e664fd11f01 2020-09-08T17:12:38.851 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:53.360+0000 7fc406a09ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.851 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:53.362+0000 7fc406a09ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.851 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:53.363+0000 7fc406a09ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.851 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:53.364+0000 7fc406a09ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:12:38.852 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:53.364+0000 7fc406a09ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:12:38.852 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:53.364+0000 7fc406a09ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:12:38.852 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:53.405+0000 7fc406a09ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:12:38.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:12:38.852 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:38.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:12:38.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:38.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:12:38.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:38.853 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:38.854 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:38.854 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:57.833+0000 7f5e28dacec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.854 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:57.835+0000 7f5e28dacec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.854 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:57.837+0000 7f5e28dacec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:12:38.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:38.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:12:38.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:38.855 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:38.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:38.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:38.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:38.856 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:12:38.856 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:58.360+0000 7f5e28dacec0 -1 Falling back to public interface 2020-09-08T17:12:38.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:38.857 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:11:59.147+0000 7f5e28dacec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:12:38.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:38.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:38.857 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:38.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:38.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:12:38.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:38.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:38.858 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:38.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:12:38.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:12:38.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:12:38.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:12:38.859 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:12:38.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:12:38.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:12:38.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:12:38.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:12:38.860 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:12:38.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:12:38.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:12:38.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:12:38.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:12:38.861 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:12:38.862 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:12:38.862 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:12:38.862 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:12:38.862 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:12:38.862 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:12:38.863 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:38.863 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:38.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:12:38.863 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:12:38.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:12:38.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:12:38.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:12:38.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:12:38.864 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:12:38.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:12:38.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:12:38.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:12:38.865 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:12:38.866 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:12:38.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5d50328b-a024-43bc-bcb7-8c7cc165e7ae 2020-09-08T17:12:38.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 5d50328b-a024-43bc-bcb7-8c7cc165e7ae' 2020-09-08T17:12:38.866 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:12:38.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBfu1df96GiNxAAOyjV40iGAES2Apo4c6Sf/Q== 2020-09-08T17:12:38.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBfu1df96GiNxAAOyjV40iGAES2Apo4c6Sf/Q=="}' 2020-09-08T17:12:38.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5d50328b-a024-43bc-bcb7-8c7cc165e7ae -i td/test-erasure-eio/3/new.json 2020-09-08T17:12:38.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:12:38.867 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBfu1df96GiNxAAOyjV40iGAES2Apo4c6Sf/Q== --osd-uuid 5d50328b-a024-43bc-bcb7-8c7cc165e7ae 2020-09-08T17:12:38.867 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:00.841+0000 7f67549acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.868 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:00.875+0000 7f67549acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.868 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:00.877+0000 7f67549acec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:38.868 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:00.877+0000 7f67549acec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:12:38.868 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:00.877+0000 7f67549acec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:12:38.868 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:00.877+0000 7f67549acec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:12:38.869 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:01.016+0000 7f67549acec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:12:38.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:12:38.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:12:38.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:12:38.869 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:12:38.870 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:12:38.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:12:38.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:12:38.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:12:38.871 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:12:39.074 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:06.285+0000 7f3213daaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:39.074 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:06.286+0000 7f3213daaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:39.074 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:06.287+0000 7f3213daaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:12:39.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:12:39.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:12:39.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:12:39.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:12:39.075 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:12:39.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:39.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:12:39.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:39.076 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:12:39.076 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:06.805+0000 7f3213daaec0 -1 Falling back to public interface 2020-09-08T17:12:39.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:12:39.077 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:07.587+0000 7f3213daaec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:12:39.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:12:39.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:12:39.077 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:12:39.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:12:39.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:12:39.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:12:39.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:12:39.078 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:12:39.079 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:12:39.079 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:12:39.079 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:12:39.079 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:12:39.079 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:12:39.080 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:12:39.080 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:12:39.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:12:39.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:12:39.080 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:12:39.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:269: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2020-09-08T17:12:39.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:270: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2020-09-08T17:12:39.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:12:39.081 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:12:39.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=2 2020-09-08T17:12:39.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:12:39.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=1 2020-09-08T17:12:39.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:12:39.082 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2020-09-08T17:12:39.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:12:39.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:12:39.083 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:12:39.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:12:39.083 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:12:39.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:12:39.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:12:39.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:12:39.084 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:12:39.084 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:12:39.085 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:12:39.085 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:12:39.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:12:39.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:12:39.085 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:12:39.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:12:39.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:12:39.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:12:39.086 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:12:39.086 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:12:39.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:12:39.087 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:12:39.087 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:12:39.087 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:12:39.087 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:12:39.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:39.088 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:12:39.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803785 2020-09-08T17:12:39.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803785 2020-09-08T17:12:39.088 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785' 2020-09-08T17:12:39.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:39.089 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:12:39.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607558 2020-09-08T17:12:39.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607558 2020-09-08T17:12:39.089 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607558' 2020-09-08T17:12:39.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:39.090 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:12:39.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411333 2020-09-08T17:12:39.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411333 2020-09-08T17:12:39.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607558 2-77309411333' 2020-09-08T17:12:39.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:12:39.091 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:12:39.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149699 2020-09-08T17:12:39.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149699 2020-09-08T17:12:39.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607558 2-77309411333 3-111669149699' 2020-09-08T17:12:39.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:12:39.092 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803785 2020-09-08T17:12:39.092 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:12:39.092 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:12:39.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803785 2020-09-08T17:12:39.294 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:12:39.294 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803785 2020-09-08T17:12:39.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2020-09-08T17:12:39.295 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:12:39.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803785 -lt 25769803785 2020-09-08T17:12:39.295 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:12:39.295 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607558 2020-09-08T17:12:39.296 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:12:39.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:12:39.296 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607558 2020-09-08T17:12:39.296 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:12:39.296 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607558 2020-09-08T17:12:39.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607558' 2020-09-08T17:12:39.297 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:12:39.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607558 -lt 51539607558 2020-09-08T17:12:39.297 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:12:39.297 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411333 2020-09-08T17:12:39.298 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:12:39.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:12:39.298 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411333 2020-09-08T17:12:39.298 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:12:39.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411333 2020-09-08T17:12:39.502 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411333' 2020-09-08T17:13:22.362 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:13:22.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411333 -lt 77309411333 2020-09-08T17:13:23.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:13:23.182 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149699 2020-09-08T17:13:23.182 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:13:23.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:13:23.183 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149699 2020-09-08T17:13:23.183 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:13:23.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149699 2020-09-08T17:13:23.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2020-09-08T17:13:23.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:13:23.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149699 -lt 111669149699 2020-09-08T17:13:23.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:13:23.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:13:23.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:13:23.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:13:23.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:13:23.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:13:23.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:13:23.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:13:23.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:13:23.390 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:13:25.430 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:13:25.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:13:25.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:13:25.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:13:25.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:13:25.431 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:13:25.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:13:25.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:13:25.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:272: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2020-09-08T17:13:25.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:273: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2020-09-08T17:13:25.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:146: rados_put_get_data: local inject=eio 2020-09-08T17:13:25.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: shift 2020-09-08T17:13:25.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: local dir=td/test-erasure-eio 2020-09-08T17:13:25.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: shift 2020-09-08T17:13:25.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: local shard_id=1 2020-09-08T17:13:25.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: shift 2020-09-08T17:13:25.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: local arg= 2020-09-08T17:13:25.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:156: rados_put_get_data: local poolname=pool-jerasure 2020-09-08T17:13:25.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local objname=obj-eio-59146-1 2020-09-08T17:13:25.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-59146-1 2020-09-08T17:13:25.638 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:13:40.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:13:40.326 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-eio-59146-1 2020-09-08T17:13:40.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:13:40.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:13:40.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:13:40.327 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:13:40.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:13:40.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:13:40.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:13:40.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:13:40.328 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-eio-59146-1 td/test-erasure-eio/ORIGINAL 2020-09-08T17:13:40.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-59146-1 td/test-erasure-eio 1 2020-09-08T17:13:40.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:13:40.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:13:40.329 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:13:40.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:13:40.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:13:40.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:13:40.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-59146-1 2020-09-08T17:13:40.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:13:40.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:13:40.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:13:40.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2020-09-08T17:13:40.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:13:40.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:13:40.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj-eio-59146-1 2020-09-08T17:13:40.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:13:40.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-59146-1 2020-09-08T17:13:40.332 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-eio-59146-1 2020-09-08T17:13:40.332 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:13:40.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:13:40.333 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:13:40.333 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:13:40.333 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:13:40.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:13:40.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2020-09-08T17:13:40.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:13:40.334 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/1/type 2020-09-08T17:13:40.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:13:40.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-09-08T17:13:40.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:13:40.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2020-09-08T17:13:40.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:13:40.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:13:40.538 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:13:40.538 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2020-09-08T17:13:40.538 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T17:13:40.539 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T17:13:40.539 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:13:40.539 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:13:40.539 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:13:40.540 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T17:13:40.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:13:40.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:13:40.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:13:40.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:13:40.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2020-09-08T17:13:40.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-09-08T17:13:40.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-09-08T17:13:40.541 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:13:40.541 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:13:40.542 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:13:40.542 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.1.asok 2020-09-08T17:13:40.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:13:40.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-59146-1 1 2020-09-08T17:13:40.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-59146-1 2020-09-08T17:13:40.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:13:40.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:13:40.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-eio-59146-1 2020-09-08T17:13:40.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:13:40.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:13:40.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-eio-59146-1 td/test-erasure-eio/COPY 2020-09-08T17:13:40.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:13:40.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:13:40.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:162: rados_put_get_data: '[' '' = recovery ']' 2020-09-08T17:13:40.748 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:180: rados_put_get_data: expr 1 + 1 2020-09-08T17:13:40.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:180: rados_put_get_data: shard_id=2 2020-09-08T17:13:40.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-59146-1 td/test-erasure-eio 2 2020-09-08T17:13:40.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=ec 2020-09-08T17:13:40.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2020-09-08T17:13:40.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2020-09-08T17:13:40.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2020-09-08T17:13:40.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-jerasure 2020-09-08T17:13:40.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2020-09-08T17:13:40.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-59146-1 2020-09-08T17:13:40.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2020-09-08T17:13:40.750 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/test-erasure-eio 2020-09-08T17:13:40.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2020-09-08T17:13:40.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=2 2020-09-08T17:13:40.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2020-09-08T17:13:40.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:13:40.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-jerasure obj-eio-59146-1 2020-09-08T17:13:40.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:13:40.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-59146-1 2020-09-08T17:13:40.955 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-eio-59146-1 2020-09-08T17:13:40.956 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:13:40.956 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:13:40.956 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:13:40.956 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:13:40.956 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:13:40.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2020-09-08T17:13:40.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=2 2020-09-08T17:13:40.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' ec '!=' ec ']' 2020-09-08T17:13:40.957 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/test-erasure-eio/2/type 2020-09-08T17:13:40.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2020-09-08T17:13:40.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2020-09-08T17:13:40.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2020-09-08T17:13:40.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=2 2020-09-08T17:13:40.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2020-09-08T17:13:40.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2020-09-08T17:13:40.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2020-09-08T17:13:41.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.2 2020-09-08T17:13:41.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-09-08T17:13:41.166 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-09-08T17:13:41.167 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:13:41.167 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:13:41.167 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:13:41.167 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.2.asok 2020-09-08T17:13:41.167 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.59146/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2020-09-08T17:13:41.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2020-09-08T17:13:41.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2020-09-08T17:13:41.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2020-09-08T17:13:41.168 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.2 2020-09-08T17:13:41.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-09-08T17:13:41.169 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-09-08T17:13:41.169 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:13:41.169 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:13:41.169 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:13:41.170 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.2.asok 2020-09-08T17:13:41.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2020-09-08T17:13:41.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-59146-1 2 2020-09-08T17:13:41.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-59146-1 fail 2020-09-08T17:13:41.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:13:41.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:13:41.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-eio-59146-1 2020-09-08T17:13:41.578 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:13:41.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:13:41.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-eio-59146-1 td/test-erasure-eio/COPY 2020-09-08T17:13:41.579 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-eio-59146-1: (5) Input/output error 2020-09-08T17:13:41.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:13:41.579 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:13:41.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:274: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:13:41.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:13:41.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:13:41.580 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:13:41.580 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:13:41.581 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:13:41.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:13:41.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:13:41.581 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:13:41.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:13:41.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:13:41.582 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:13:41.582 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:13:41.786 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:14:04.234 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:14:04.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:14:04.235 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:14:04.235 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:14:04.235 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:14:04.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:14:04.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:14:04.236 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:14:04.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:14:04.236 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:14:04.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:14:04.237 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:14:04.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:14:04.237 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:14:04.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:14:04.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:14:04.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:04.238 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:04.238 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:14:04.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:14:04.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:14:04.239 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:14:04.442 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:14:04.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:14:04.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:14:04.646 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:14:04.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:14:04.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:14:04.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:14:04.647 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:14:04.647 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:14:04.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:14:04.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:14:04.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:14:04.648 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:14:04.648 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:14:04.649 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:14:04.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:14:04.649 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:14:04.649 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:14:04.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:14:04.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:14:04.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:14:04.650 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:14:04.650 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:14:04.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:14:04.651 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:14:04.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:14:04.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:04.651 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:04.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:14:04.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:14:04.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:14:04.652 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:14:04.652 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:14:04.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:04.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:04.653 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.59146 2020-09-08T17:14:04.653 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-09-08T17:14:04.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-09-08T17:14:04.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-09-08T17:14:04.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2020-09-08T17:14:04.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: run_mon td/test-erasure-eio a 2020-09-08T17:14:04.654 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2020-09-08T17:14:04.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-09-08T17:14:04.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-09-08T17:14:04.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-09-08T17:14:04.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2020-09-08T17:14:04.655 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2020-09-08T17:14:04.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-09-08T17:14:04.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:14:04.656 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:14:04.656 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:14:04.656 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:04.657 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:04.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:04.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-09-08T17:14:04.657 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2020-09-08T17:14:04.657 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2020-09-08T17:14:04.658 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:14:04.658 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:14:04.658 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2020-09-08T17:14:04.658 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2020-09-08T17:14:04.658 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:14:04.659 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:14:04.659 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:14:04.659 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:14:04.659 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:04.862 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:04.862 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:14:04.862 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:14:04.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get fsid 2020-09-08T17:14:04.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2020-09-08T17:14:04.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2020-09-08T17:14:04.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2020-09-08T17:14:04.863 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2020-09-08T17:14:04.864 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2020-09-08T17:14:04.864 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2020-09-08T17:14:04.864 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:14:04.864 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:14:04.864 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:14:04.865 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:04.865 INFO:tasks.workunit.client.0.smithi120.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:04.865 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:14:04.865 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2020-09-08T17:14:04.866 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.59146/ceph-mon.a.asok config get mon_host 2020-09-08T17:14:04.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mgr td/test-erasure-eio x 2020-09-08T17:14:04.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/test-erasure-eio 2020-09-08T17:14:04.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2020-09-08T17:14:04.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2020-09-08T17:14:05.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2020-09-08T17:14:05.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/test-erasure-eio/x 2020-09-08T17:14:05.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-09-08T17:14:05.070 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2020-09-08T17:14:05.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:14:05.071 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:14:05.071 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:14:05.071 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:05.072 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:05.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:05.072 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:14:05.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-09-08T17:14:05.072 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: create_pool rbd 4 2020-09-08T17:14:05.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2020-09-08T17:14:05.073 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'rbd' already exists 2020-09-08T17:14:05.073 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:14:05.073 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: get_asok_path mon.a 2020-09-08T17:14:05.073 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-09-08T17:14:05.074 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-09-08T17:14:05.074 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:14:05.074 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:05.278 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:05.278 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-mon.a.asok 2020-09-08T17:14:05.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: CEPH_ARGS= 2020-09-08T17:14:05.278 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:38: run: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-mon.a.asok log flush 2020-09-08T17:14:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2020-09-08T17:14:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: TEST_rados_get_subread_missing td/test-erasure-eio 2020-09-08T17:14:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:281: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2020-09-08T17:14:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_missing: setup_osds 4 2020-09-08T17:14:05.279 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:46: setup_osds: local count=4 2020-09-08T17:14:05.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: shift 2020-09-08T17:14:05.280 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: expr 4 - 1 2020-09-08T17:14:05.280 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: seq 0 3 2020-09-08T17:14:05.280 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:14:05.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 0 2020-09-08T17:14:05.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:14:05.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:14:05.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2020-09-08T17:14:05.281 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:14:05.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/0 2020-09-08T17:14:05.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:14:05.282 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:14:05.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:14:05.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:14:05.486 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2020-09-08T17:14:05.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2020-09-08T17:14:05.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:14:05.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:14:05.487 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:14:05.487 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:14:05.488 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:14:05.488 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:14:05.488 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:14:05.488 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:05.488 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:05.489 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:05.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:05.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:14:05.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:14:05.489 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:14:05.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:14:05.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:14:05.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:14:05.490 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:14:05.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:14:05.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:14:05.694 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:14:05.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/0 2020-09-08T17:14:05.695 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:14:05.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2262a336-c077-4c99-960b-d58e72c3f350 2020-09-08T17:14:05.695 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 2262a336-c077-4c99-960b-d58e72c3f350' 2020-09-08T17:14:05.696 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:14:05.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB0u1dfDqvvJxAAO2ie3LpE3S7FaY4levZWlw== 2020-09-08T17:14:05.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB0u1dfDqvvJxAAO2ie3LpE3S7FaY4levZWlw=="}' 2020-09-08T17:14:05.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2262a336-c077-4c99-960b-d58e72c3f350 -i td/test-erasure-eio/0/new.json 2020-09-08T17:14:05.696 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/0/new.json 2020-09-08T17:14:05.697 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB0u1dfDqvvJxAAO2ie3LpE3S7FaY4levZWlw== --osd-uuid 2262a336-c077-4c99-960b-d58e72c3f350 2020-09-08T17:14:05.697 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:21.021+0000 7f639fd37ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:05.697 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:21.023+0000 7f639fd37ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:05.697 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:21.024+0000 7f639fd37ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:05.697 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:21.025+0000 7f639fd37ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:14:05.906 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:21.025+0000 7f639fd37ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:14:06.114 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:21.025+0000 7f639fd37ec0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2020-09-08T17:14:06.114 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:21.073+0000 7f639fd37ec0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2020-09-08T17:14:06.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2020-09-08T17:14:06.114 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:14:06.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2020-09-08T17:14:06.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:14:06.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2020-09-08T17:14:06.115 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:14:06.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:14:06.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:14:06.116 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:14:06.116 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:25.528+0000 7fd5a1e99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:06.116 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:25.530+0000 7fd5a1e99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:06.117 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:25.531+0000 7fd5a1e99ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:06.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2020-09-08T17:14:06.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:14:06.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2020-09-08T17:14:06.117 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:14:06.322 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:14:17.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:14:17.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:14:17.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:14:17.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:14:17.564 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:26.047+0000 7fd5a1e99ec0 -1 Falling back to public interface 2020-09-08T17:14:17.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:14:17.564 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:26.814+0000 7fd5a1e99ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-09-08T17:14:17.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:14:17.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:14:17.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:14:17.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:14:17.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2020-09-08T17:14:17.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:14:17.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:14:17.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:14:17.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:14:17.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 1 2020-09-08T17:14:17.568 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:14:17.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:14:17.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2020-09-08T17:14:17.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:14:17.569 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:14:17.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:14:17.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:14:17.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:14:17.570 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:14:17.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:14:17.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:14:17.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:14:17.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:14:17.571 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:14:17.572 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:14:17.572 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:14:17.572 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:14:17.572 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:14:17.572 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:17.573 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:17.573 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:17.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:17.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:14:17.573 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:14:17.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:14:17.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:14:17.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:14:17.574 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:14:17.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:14:17.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:14:17.778 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:14:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:14:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:14:17.779 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:14:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=32e5f7f8-49c7-49e3-a5e2-e923aaece645 2020-09-08T17:14:17.779 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 32e5f7f8-49c7-49e3-a5e2-e923aaece645' 2020-09-08T17:14:17.780 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:14:17.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB7u1df/76OKBAA53sQ1drrVbyaJi53SYYB3g== 2020-09-08T17:14:17.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB7u1df/76OKBAA53sQ1drrVbyaJi53SYYB3g=="}' 2020-09-08T17:14:17.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 32e5f7f8-49c7-49e3-a5e2-e923aaece645 -i td/test-erasure-eio/1/new.json 2020-09-08T17:14:17.780 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/1/new.json 2020-09-08T17:14:17.781 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB7u1df/76OKBAA53sQ1drrVbyaJi53SYYB3g== --osd-uuid 32e5f7f8-49c7-49e3-a5e2-e923aaece645 2020-09-08T17:14:17.781 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:28.239+0000 7fb829cd8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:17.781 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:28.249+0000 7fb829cd8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:17.781 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:28.249+0000 7fb829cd8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:17.986 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:28.250+0000 7fb829cd8ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:14:18.598 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:28.250+0000 7fb829cd8ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:14:18.598 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:28.250+0000 7fb829cd8ec0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2020-09-08T17:14:18.598 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:28.332+0000 7fb829cd8ec0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2020-09-08T17:14:18.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2020-09-08T17:14:18.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:14:18.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2020-09-08T17:14:18.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:14:18.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2020-09-08T17:14:18.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:14:18.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:14:18.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:14:18.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:14:18.600 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:32.788+0000 7f25c9f71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:18.601 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:32.790+0000 7f25c9f71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:18.601 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:32.791+0000 7f25c9f71ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:18.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2020-09-08T17:14:18.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:14:18.802 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:14:37.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:14:37.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:14:37.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:14:37.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:14:37.167 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:14:37.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:14:37.168 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:33.306+0000 7f25c9f71ec0 -1 Falling back to public interface 2020-09-08T17:14:37.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:14:37.168 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:34.073+0000 7f25c9f71ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-09-08T17:14:37.168 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:14:37.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:14:37.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:14:37.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:14:37.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:14:37.169 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:14:37.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:14:37.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:14:37.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:14:37.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 2 2020-09-08T17:14:37.170 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:14:37.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:14:37.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2020-09-08T17:14:37.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:14:37.171 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:14:37.374 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:14:41.866 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:14:42.070 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:14:43.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:14:43.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:14:43.090 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:14:43.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:14:43.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:14:43.091 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:14:43.091 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:14:43.092 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:14:43.092 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:14:43.092 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:14:43.092 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:43.092 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:43.093 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:43.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:43.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:14:43.093 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:14:43.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:14:43.094 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:14:43.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:14:43.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:14:43.298 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:14:43.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:14:43.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:14:43.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:14:43.299 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:14:43.299 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:14:43.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=450871d5-a0a3-4f4a-ab84-960c5999c21f 2020-09-08T17:14:43.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 450871d5-a0a3-4f4a-ab84-960c5999c21f' 2020-09-08T17:14:43.300 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:14:43.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCCu1dfwcxKORAAqwWUjAH61ISb5l5yZQQq1g== 2020-09-08T17:14:43.300 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCCu1dfwcxKORAAqwWUjAH61ISb5l5yZQQq1g=="}' 2020-09-08T17:14:43.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 450871d5-a0a3-4f4a-ab84-960c5999c21f -i td/test-erasure-eio/2/new.json 2020-09-08T17:14:43.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/2/new.json 2020-09-08T17:14:43.301 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCCu1dfwcxKORAAqwWUjAH61ISb5l5yZQQq1g== --osd-uuid 450871d5-a0a3-4f4a-ab84-960c5999c21f 2020-09-08T17:14:43.301 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:35.581+0000 7f5f13748ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:43.301 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:35.583+0000 7f5f13748ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:43.506 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:35.583+0000 7f5f13748ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:43.506 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:35.584+0000 7f5f13748ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:14:43.506 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:35.584+0000 7f5f13748ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:14:43.507 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:35.584+0000 7f5f13748ec0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2020-09-08T17:14:43.507 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:35.652+0000 7f5f13748ec0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2020-09-08T17:14:43.508 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 51539607559 2020-09-08T17:14:43.508 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411333 2020-09-08T17:14:43.508 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149699 2020-09-08T17:14:43.508 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:14:43.509 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:11:35.828+0000 7fd287a946c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:14:43.509 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 8e876070-781c-4073-ae25-749f1e81ebb9 2020-09-08T17:14:43.509 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:43.509 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:14:43.509 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:14:43.510 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:43.510 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:43.510 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174064,v1:127.0.0.1:6803/174064] [v2:127.0.0.1:6804/174064,v1:127.0.0.1:6805/174064] exists,up 8e876070-781c-4073-ae25-749f1e81ebb9 2020-09-08T17:14:43.510 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 3e0da9c2-1703-49be-868e-b0263a7c279e 2020-09-08T17:14:43.511 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:43.511 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:14:43.511 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:14:43.511 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:43.511 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:43.512 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/175058,v1:127.0.0.1:6811/175058] [v2:127.0.0.1:6812/175058,v1:127.0.0.1:6813/175058] exists,up 3e0da9c2-1703-49be-868e-b0263a7c279e 2020-09-08T17:14:43.512 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 7d5cf1c2-2f69-4487-8203-9e664fd11f01 2020-09-08T17:14:43.512 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:14:43.512 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd2 key to auth repository 2020-09-08T17:14:43.512 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.2 2020-09-08T17:14:43.513 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:43.513 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:43.513 INFO:tasks.workunit.client.0.smithi120.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/176052,v1:127.0.0.1:6819/176052] [v2:127.0.0.1:6820/176052,v1:127.0.0.1:6821/176052] exists,up 7d5cf1c2-2f69-4487-8203-9e664fd11f01 2020-09-08T17:14:43.714 INFO:tasks.workunit.client.0.smithi120.stdout:add osd3 5d50328b-a024-43bc-bcb7-8c7cc165e7ae 2020-09-08T17:14:47.390 INFO:tasks.workunit.client.0.smithi120.stdout:3 2020-09-08T17:14:47.390 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd3 key to auth repository 2020-09-08T17:14:47.390 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.3 2020-09-08T17:14:47.391 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:47.391 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:47.391 INFO:tasks.workunit.client.0.smithi120.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/177046,v1:127.0.0.1:6827/177046] [v2:127.0.0.1:6828/177046,v1:127.0.0.1:6829/177046] exists,up 5d50328b-a024-43bc-bcb7-8c7cc165e7ae 2020-09-08T17:14:47.391 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:14:47.392 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:11:43.818+0000 7f607a3c5ec0 0 load: jerasure load: lrc load: isa 2020-09-08T17:14:47.392 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.0 seq 25769803785 2020-09-08T17:14:59.026 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.1 seq 51539607558 2020-09-08T17:14:59.026 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.2 seq 77309411333 2020-09-08T17:14:59.026 INFO:tasks.workunit.client.0.smithi120.stdout:waiting osd.3 seq 111669149699 2020-09-08T17:14:59.027 INFO:tasks.workunit.client.0.smithi120.stdout:{} 2020-09-08T17:14:59.027 INFO:tasks.workunit.client.0.smithi120.stdout:2020-09-08T17:12:18.066+0000 7f3451d7e6c0 0 load: jerasure load: lrc load: isa 2020-09-08T17:14:59.027 INFO:tasks.workunit.client.0.smithi120.stdout:add osd0 2262a336-c077-4c99-960b-d58e72c3f350 2020-09-08T17:14:59.027 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:59.028 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd0 key to auth repository 2020-09-08T17:14:59.028 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.0 2020-09-08T17:14:59.028 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:59.028 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:59.028 INFO:tasks.workunit.client.0.smithi120.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/179025,v1:127.0.0.1:6803/179025] [v2:127.0.0.1:6804/179025,v1:127.0.0.1:6805/179025] exists,up 2262a336-c077-4c99-960b-d58e72c3f350 2020-09-08T17:14:59.029 INFO:tasks.workunit.client.0.smithi120.stdout:add osd1 32e5f7f8-49c7-49e3-a5e2-e923aaece645 2020-09-08T17:14:59.029 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:59.029 INFO:tasks.workunit.client.0.smithi120.stdout:adding osd1 key to auth repository 2020-09-08T17:14:59.029 INFO:tasks.workunit.client.0.smithi120.stdout:start osd.1 2020-09-08T17:14:59.029 INFO:tasks.workunit.client.0.smithi120.stdout:0 2020-09-08T17:14:59.029 INFO:tasks.workunit.client.0.smithi120.stdout:1 2020-09-08T17:14:59.030 INFO:tasks.workunit.client.0.smithi120.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/180019,v1:127.0.0.1:6811/180019] [v2:127.0.0.1:6812/180019,v1:127.0.0.1:6813/180019] exists,up 32e5f7f8-49c7-49e3-a5e2-e923aaece645 2020-09-08T17:14:59.030 INFO:tasks.workunit.client.0.smithi120.stdout:add osd2 450871d5-a0a3-4f4a-ab84-960c5999c21f 2020-09-08T17:14:59.030 INFO:tasks.workunit.client.0.smithi120.stdout:2 2020-09-08T17:14:59.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2020-09-08T17:14:59.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:14:59.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2020-09-08T17:14:59.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:14:59.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2020-09-08T17:14:59.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:14:59.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:14:59.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:14:59.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:14:59.212 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:40.099+0000 7f5d3f528ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:59.212 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:40.101+0000 7f5d3f528ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:40.103+0000 7f5d3f528ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:14:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2020-09-08T17:14:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:14:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:14:59.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:14:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:14:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:14:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:14:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:14:59.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:14:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:40.620+0000 7f5d3f528ec0 -1 Falling back to public interface 2020-09-08T17:14:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:14:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:41.469+0000 7f5d3f528ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-09-08T17:14:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:14:59.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:14:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:14:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:14:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:14:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:14:59.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:14:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:14:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:49: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2020-09-08T17:14:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: run_osd td/test-erasure-eio 3 2020-09-08T17:14:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/test-erasure-eio 2020-09-08T17:14:59.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2020-09-08T17:14:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2020-09-08T17:14:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2020-09-08T17:14:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/test-erasure-eio/3 2020-09-08T17:14:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:14:59.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:14:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:14:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:14:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2020-09-08T17:14:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2020-09-08T17:14:59.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2020-09-08T17:14:59.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2020-09-08T17:14:59.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:14:59.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2020-09-08T17:14:59.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:14:59.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:14:59.423 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:14:59.423 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:14:59.424 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:14:59.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:59.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:14:59.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:14:59.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2020-09-08T17:14:59.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2020-09-08T17:14:59.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:14:59.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:14:59.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:14:59.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:14:59.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:14:59.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2020-09-08T17:14:59.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2020-09-08T17:14:59.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/test-erasure-eio/3 2020-09-08T17:14:59.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2020-09-08T17:14:59.630 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c785feb8-3fe4-4741-a228-1e502e991db2 2020-09-08T17:15:03.922 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 c785feb8-3fe4-4741-a228-1e502e991db2' 2020-09-08T17:15:03.922 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2020-09-08T17:15:03.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCKu1dfI0m1DxAALkdIJdxAATIfQW1Ec08q7A== 2020-09-08T17:15:03.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCKu1dfI0m1DxAALkdIJdxAATIfQW1Ec08q7A=="}' 2020-09-08T17:15:03.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c785feb8-3fe4-4741-a228-1e502e991db2 -i td/test-erasure-eio/3/new.json 2020-09-08T17:15:03.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/test-erasure-eio/3/new.json 2020-09-08T17:15:03.923 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCKu1dfI0m1DxAALkdIJdxAATIfQW1Ec08q7A== --osd-uuid c785feb8-3fe4-4741-a228-1e502e991db2 2020-09-08T17:15:03.924 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:43.128+0000 7f3456fd2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:03.924 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:43.171+0000 7f3456fd2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:03.924 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:43.172+0000 7f3456fd2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:03.924 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:43.173+0000 7f3456fd2ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:15:03.924 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:43.173+0000 7f3456fd2ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:15:03.925 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:43.173+0000 7f3456fd2ec0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2020-09-08T17:15:03.925 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:43.254+0000 7f3456fd2ec0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2020-09-08T17:15:03.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2020-09-08T17:15:03.925 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2020-09-08T17:15:04.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2020-09-08T17:15:04.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-09-08T17:15:04.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2020-09-08T17:15:04.330 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:15:04.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2020-09-08T17:15:04.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2020-09-08T17:15:04.331 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2020-09-08T17:15:04.331 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:48.253+0000 7f91fb9beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:04.331 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:48.265+0000 7f91fb9beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:04.332 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:48.266+0000 7f91fb9beec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:04.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2020-09-08T17:15:04.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:15:04.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2020-09-08T17:15:04.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:15:04.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:15:04.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:15:04.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:15:04.333 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:15:04.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:15:04.538 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:48.793+0000 7f91fb9beec0 -1 Falling back to public interface 2020-09-08T17:15:04.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:15:04.538 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:12:49.608+0000 7f91fb9beec0 -1 osd.3 0 log_to_monitors {default=true} 2020-09-08T17:15:04.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:15:04.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:15:04.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:15:04.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:15:04.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2020-09-08T17:15:04.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:15:04.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:15:04.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:15:04.540 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: get_asok_path osd.0 2020-09-08T17:15:04.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-09-08T17:15:04.541 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-09-08T17:15:04.541 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-09-08T17:15:04.541 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:15:04.541 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:15:04.542 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.59146/ceph-osd.0.asok 2020-09-08T17:15:04.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: CEPH_ARGS= 2020-09-08T17:15:04.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:54: setup_osds: ceph --admin-daemon /tmp/ceph-asok.59146/ceph-osd.0.asok log flush 2020-09-08T17:15:04.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2020-09-08T17:15:04.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:284: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2020-09-08T17:15:04.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:285: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2020-09-08T17:15:04.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:66: create_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:15:04.746 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: shift 2020-09-08T17:15:04.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: local k=2 2020-09-08T17:15:04.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: shift 2020-09-08T17:15:04.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: local m=1 2020-09-08T17:15:04.747 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: shift 2020-09-08T17:15:04.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:73: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2020-09-08T17:15:04.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:77: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2020-09-08T17:15:04.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2020-09-08T17:15:04.748 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' already exists 2020-09-08T17:15:04.748 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2020-09-08T17:15:04.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:79: create_erasure_coded_pool: wait_for_clean 2020-09-08T17:15:04.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:15:04.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:15:04.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:15:04.749 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:15:04.750 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:15:04.750 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:15:04.750 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:15:04.750 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:15:04.751 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:15:04.954 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:15:04.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:15:04.954 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:15:04.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:15:04.955 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:15:04.955 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:15:04.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:15:04.956 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:15:04.956 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:15:04.956 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:15:04.956 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:15:04.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:04.957 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:15:04.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803784 2020-09-08T17:15:04.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803784 2020-09-08T17:15:04.957 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784' 2020-09-08T17:15:04.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:04.958 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:15:04.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607558 2020-09-08T17:15:04.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607558 2020-09-08T17:15:04.958 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-51539607558' 2020-09-08T17:15:04.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:04.959 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:15:04.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411333 2020-09-08T17:15:04.959 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411333 2020-09-08T17:15:05.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-51539607558 2-77309411333' 2020-09-08T17:15:05.162 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:05.162 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:15:05.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149699 2020-09-08T17:15:05.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149699 2020-09-08T17:15:05.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-51539607558 2-77309411333 3-111669149699' 2020-09-08T17:15:05.163 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:05.163 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803784 2020-09-08T17:15:05.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:05.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:15:05.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803784 2020-09-08T17:15:05.164 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:05.164 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803784 2020-09-08T17:15:05.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2020-09-08T17:15:05.165 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:15:05.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803781 -lt 25769803784 2020-09-08T17:15:05.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:15:05.165 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:15:05.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:15:05.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803784 -lt 25769803784 2020-09-08T17:15:05.166 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:05.166 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607558 2020-09-08T17:15:05.370 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:07.418 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:15:07.419 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607558 2020-09-08T17:15:07.419 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:07.419 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607558 2020-09-08T17:15:07.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607558' 2020-09-08T17:15:07.420 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:15:07.420 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607558 -lt 51539607558 2020-09-08T17:15:07.421 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:07.421 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411333 2020-09-08T17:15:07.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:07.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:15:07.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411333 2020-09-08T17:15:07.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:07.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411333 2020-09-08T17:15:07.423 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411333' 2020-09-08T17:15:07.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:15:07.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411333 -lt 77309411333 2020-09-08T17:15:07.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:07.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149699 2020-09-08T17:15:07.425 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:07.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:15:07.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149699 2020-09-08T17:15:07.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:07.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149699 2020-09-08T17:15:07.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149699' 2020-09-08T17:15:07.427 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:15:07.427 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149700 -lt 111669149699 2020-09-08T17:15:07.428 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:15:07.428 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:15:07.429 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:15:07.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:15:07.429 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:15:07.430 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:15:07.430 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:15:07.430 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:15:07.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:15:07.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:15:07.431 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:15:07.432 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:15:07.432 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:15:07.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:15:07.433 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:15:07.433 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:15:07.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:15:07.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:15:07.434 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:287: TEST_rados_get_subread_missing: local shard_id=1 2020-09-08T17:15:07.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2020-09-08T17:15:07.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:146: rados_put_get_data: local inject=remove 2020-09-08T17:15:07.435 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: shift 2020-09-08T17:15:07.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: local dir=td/test-erasure-eio 2020-09-08T17:15:07.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: shift 2020-09-08T17:15:07.436 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: local shard_id=1 2020-09-08T17:15:07.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: shift 2020-09-08T17:15:07.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: local arg= 2020-09-08T17:15:07.437 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:156: rados_put_get_data: local poolname=pool-jerasure 2020-09-08T17:15:07.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local objname=obj-remove-59146-1 2020-09-08T17:15:07.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-59146-1 2020-09-08T17:15:07.438 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:89: rados_put: local dir=td/test-erasure-eio 2020-09-08T17:15:07.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local poolname=pool-jerasure 2020-09-08T17:15:07.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local objname=obj-remove-59146-1 2020-09-08T17:15:07.439 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:15:07.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 AAA 2020-09-08T17:15:07.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:15:07.440 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 BBB 2020-09-08T17:15:07.642 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:15:11.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 CCCC 2020-09-08T17:15:11.122 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:93: rados_put: for marker in AAA BBB CCCC DDDD 2020-09-08T17:15:11.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: printf '%*s' 1024 DDDD 2020-09-08T17:15:11.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:99: rados_put: rados --pool pool-jerasure put obj-remove-59146-1 td/test-erasure-eio/ORIGINAL 2020-09-08T17:15:11.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-59146-1 td/test-erasure-eio 1 2020-09-08T17:15:11.123 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:126: inject_remove: local pooltype=ec 2020-09-08T17:15:11.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: shift 2020-09-08T17:15:11.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: local which=data 2020-09-08T17:15:11.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: shift 2020-09-08T17:15:11.124 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: local poolname=pool-jerasure 2020-09-08T17:15:11.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: shift 2020-09-08T17:15:11.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: local objname=obj-remove-59146-1 2020-09-08T17:15:11.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: shift 2020-09-08T17:15:11.125 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: local dir=td/test-erasure-eio 2020-09-08T17:15:11.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: shift 2020-09-08T17:15:11.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: local shard_id=1 2020-09-08T17:15:11.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: shift 2020-09-08T17:15:11.126 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:139: inject_remove: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:15:11.127 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:139: inject_remove: get_osds pool-jerasure obj-remove-59146-1 2020-09-08T17:15:11.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:15:11.330 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-remove-59146-1 2020-09-08T17:15:11.330 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-remove-59146-1 2020-09-08T17:15:11.331 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:15:11.331 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:15:11.331 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:15:11.331 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:15:11.332 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:15:11.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:139: inject_remove: local -a initial_osds 2020-09-08T17:15:11.332 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local osd_id=1 2020-09-08T17:15:11.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-59146-1 remove 2020-09-08T17:15:11.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:15:11.334 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:15:11.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2020-09-08T17:15:11.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:15:11.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-59146-1 remove 2020-09-08T17:15:11.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:15:11.335 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:15:11.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2020-09-08T17:15:11.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:15:11.336 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2020-09-08T17:15:11.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:15:11.336 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:15:11.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:15:11.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:15:11.538 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:15:11.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:15:11.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-59146-1 remove 2020-09-08T17:15:11.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:15:11.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:15:11.539 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2020-09-08T17:15:11.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:15:11.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2020-09-08T17:15:11.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-59146-1 remove 2020-09-08T17:15:11.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2020-09-08T17:15:11.540 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:15:11.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:15:11.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2020-09-08T17:15:11.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:15:11.541 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/1 2020-09-08T17:15:11.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:15:11.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:15:11.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:15:11.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:15:11.542 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2020-09-08T17:15:11.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2020-09-08T17:15:11.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:15:11.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:15:11.543 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:15:11.543 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:15:11.544 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:15:11.544 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:15:11.544 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:15:11.544 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:15:11.544 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:15:11.545 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:15:11.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:15:11.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:15:11.545 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:15:11.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:15:11.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:15:11.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:15:11.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:15:11.546 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:15:11.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:15:11.547 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/1 2020-09-08T17:15:11.548 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2020-09-08T17:15:11.549 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.1 2020-09-08T17:15:11.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:15:11.549 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/1/whoami 2020-09-08T17:15:11.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2020-09-08T17:15:11.549 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:15:11.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:15:11.550 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:15:11.550 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:01.234+0000 7f5dcfbffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:11.550 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:01.236+0000 7f5dcfbffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:11.550 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:01.237+0000 7f5dcfbffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:11.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2020-09-08T17:15:11.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:15:11.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2020-09-08T17:15:11.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:15:11.551 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:15:11.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:15:11.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:15:11.552 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:15:11.552 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:15:11.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:15:11.553 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:01.753+0000 7f5dcfbffec0 -1 Falling back to public interface 2020-09-08T17:15:11.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:15:11.553 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:02.622+0000 7f5dcfbffec0 -1 osd.1 34 log_to_monitors {default=true} 2020-09-08T17:15:11.553 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:15:11.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:15:11.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:15:11.554 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:15:11.554 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:15:11.558 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2020-09-08T17:15:11.558 INFO:tasks.workunit.client.0.smithi120.stderr:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 35 last_clean_interval [12,34) [v2:127.0.0.1:6810/183240,v1:127.0.0.1:6811/183240] [v2:127.0.0.1:6812/183240,v1:127.0.0.1:6813/183240] exists,up 32e5f7f8-49c7-49e3-a5e2-e923aaece645 2020-09-08T17:15:11.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:15:11.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:15:11.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:15:11.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:15:11.559 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:15:11.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:15:11.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:15:11.560 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:15:11.560 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:15:11.560 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:15:11.561 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:15:11.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:15:11.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:15:11.561 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:15:11.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:15:11.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:15:11.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:15:11.562 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:15:11.562 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:15:11.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:15:11.563 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:15:11.563 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:15:11.563 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:15:11.563 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:15:11.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:11.564 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:15:11.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803788 2020-09-08T17:15:11.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803788 2020-09-08T17:15:11.564 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788' 2020-09-08T17:15:11.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:11.565 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:15:11.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789955 2020-09-08T17:15:11.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789955 2020-09-08T17:15:11.565 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-158913789955' 2020-09-08T17:15:11.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:11.566 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:15:11.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=77309411337 2020-09-08T17:15:11.566 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 77309411337 2020-09-08T17:15:11.770 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-158913789955 2-77309411337' 2020-09-08T17:15:12.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:15:12.178 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:15:12.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149703 2020-09-08T17:15:12.178 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149703 2020-09-08T17:15:12.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-158913789955 2-77309411337 3-111669149703' 2020-09-08T17:15:12.179 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:12.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803788 2020-09-08T17:15:12.179 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:12.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:15:12.180 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803788 2020-09-08T17:15:12.180 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:12.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803788 2020-09-08T17:15:12.180 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2020-09-08T17:15:12.181 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803788 2020-09-08T17:15:12.181 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:15:12.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803785 -lt 25769803788 2020-09-08T17:15:12.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:15:12.181 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:15:12.182 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:15:12.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803788 2020-09-08T17:15:12.182 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:12.182 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-158913789955 2020-09-08T17:15:12.182 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:12.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:15:12.183 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-158913789955 2020-09-08T17:15:12.183 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:12.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789955 2020-09-08T17:15:12.183 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 158913789955' 2020-09-08T17:15:12.184 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 158913789955 2020-09-08T17:15:12.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:15:12.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789955 -lt 158913789955 2020-09-08T17:15:12.184 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:12.184 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-77309411337 2020-09-08T17:15:12.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:12.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:15:12.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-77309411337 2020-09-08T17:15:12.185 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:12.185 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=77309411337 2020-09-08T17:15:12.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 77309411337' 2020-09-08T17:15:12.186 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 77309411337 2020-09-08T17:15:12.186 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:15:12.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 77309411337 -lt 77309411337 2020-09-08T17:15:12.186 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:15:12.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149703 2020-09-08T17:15:12.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:15:12.187 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:15:12.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149703 2020-09-08T17:15:12.187 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:15:12.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149703 2020-09-08T17:15:12.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149703' 2020-09-08T17:15:12.188 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149703 2020-09-08T17:15:12.188 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:15:12.188 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149704 -lt 111669149703 2020-09-08T17:15:12.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:15:12.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:15:12.189 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:15:12.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:15:12.189 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:15:12.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:15:12.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:15:12.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:15:12.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:15:12.190 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:15:12.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:15:12.191 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:15:12.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:15:12.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:15:12.191 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:15:12.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:15:12.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:15:12.394 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:15:12.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-59146-1 2020-09-08T17:15:12.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:15:12.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:15:12.395 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-remove-59146-1 2020-09-08T17:15:12.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=ok 2020-09-08T17:15:12.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' ok = fail ']' 2020-09-08T17:15:12.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:119: rados_get: rados --pool pool-jerasure get obj-remove-59146-1 td/test-erasure-eio/COPY 2020-09-08T17:15:12.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2020-09-08T17:15:12.396 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: rm td/test-erasure-eio/COPY 2020-09-08T17:15:12.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:162: rados_put_get_data: '[' '' = recovery ']' 2020-09-08T17:15:12.397 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:180: rados_put_get_data: expr 1 + 1 2020-09-08T17:15:12.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:180: rados_put_get_data: shard_id=2 2020-09-08T17:15:12.397 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-59146-1 td/test-erasure-eio 2 2020-09-08T17:15:12.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:126: inject_remove: local pooltype=ec 2020-09-08T17:15:12.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: shift 2020-09-08T17:15:12.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: local which=data 2020-09-08T17:15:12.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: shift 2020-09-08T17:15:12.398 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: local poolname=pool-jerasure 2020-09-08T17:15:12.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: shift 2020-09-08T17:15:12.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: local objname=obj-remove-59146-1 2020-09-08T17:15:12.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: shift 2020-09-08T17:15:12.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: local dir=td/test-erasure-eio 2020-09-08T17:15:12.399 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: shift 2020-09-08T17:15:12.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: local shard_id=2 2020-09-08T17:15:12.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: shift 2020-09-08T17:15:12.400 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:139: inject_remove: initial_osds=($(get_osds $poolname $objname)) 2020-09-08T17:15:12.400 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:139: inject_remove: get_osds pool-jerasure obj-remove-59146-1 2020-09-08T17:15:12.400 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-jerasure 2020-09-08T17:15:12.401 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-remove-59146-1 2020-09-08T17:15:12.401 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-jerasure obj-remove-59146-1 2020-09-08T17:15:12.401 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2020-09-08T17:15:12.401 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2020-09-08T17:15:12.401 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:15:12.402 INFO:tasks.workunit.client.0.smithi120.stderr:2' 2020-09-08T17:15:12.402 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 2 2020-09-08T17:15:12.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:139: inject_remove: local -a initial_osds 2020-09-08T17:15:12.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local osd_id=2 2020-09-08T17:15:12.402 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-59146-1 remove 2020-09-08T17:15:12.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/test-erasure-eio 2020-09-08T17:15:12.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2020-09-08T17:15:12.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=2 2020-09-08T17:15:12.403 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2020-09-08T17:15:12.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-59146-1 remove 2020-09-08T17:15:12.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2020-09-08T17:15:12.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2020-09-08T17:15:12.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2020-09-08T17:15:12.404 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2020-09-08T17:15:12.405 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2020-09-08T17:15:12.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:15:12.405 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:15:12.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:15:12.406 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:15:12.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:15:12.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:15:12.407 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-59146-1 remove 2020-09-08T17:15:12.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2020-09-08T17:15:12.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2020-09-08T17:15:12.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2020-09-08T17:15:12.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2020-09-08T17:15:12.408 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2020-09-08T17:15:12.409 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-59146-1 remove 2020-09-08T17:15:12.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2020-09-08T17:15:12.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/test-erasure-eio 2020-09-08T17:15:12.610 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2020-09-08T17:15:12.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2020-09-08T17:15:12.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2020-09-08T17:15:12.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/test-erasure-eio/2 2020-09-08T17:15:12.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 ' 2020-09-08T17:15:12.611 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-09-08T17:15:12.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-09-08T17:15:12.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-09-08T17:15:12.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2020-09-08T17:15:12.612 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2020-09-08T17:15:12.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2020-09-08T17:15:12.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2020-09-08T17:15:12.613 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2020-09-08T17:15:12.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2020-09-08T17:15:12.613 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-09-08T17:15:12.614 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-09-08T17:15:12.614 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-09-08T17:15:12.614 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:15:12.614 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:15:12.614 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:15:12.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' 2020-09-08T17:15:12.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2020-09-08T17:15:12.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2020-09-08T17:15:12.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2020-09-08T17:15:12.615 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-09-08T17:15:12.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-09-08T17:15:12.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:15:12.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2020-09-08T17:15:12.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2020-09-08T17:15:12.616 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/test-erasure-eio/2 2020-09-08T17:15:12.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2020-09-08T17:15:12.617 INFO:tasks.workunit.client.0.smithi120.stderr:start osd.2 2020-09-08T17:15:12.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=12322aae-1af9-4155-a930-c3e2fe8dc619 --auth-supported=none --mon-host=127.0.0.1:7112 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.59146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-09-08T17:15:12.617 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/test-erasure-eio/2/whoami 2020-09-08T17:15:12.617 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2020-09-08T17:15:12.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2020-09-08T17:15:12.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2020-09-08T17:15:12.618 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2020-09-08T17:15:12.618 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:11.091+0000 7febb3a3bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:12.826 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:11.093+0000 7febb3a3bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:12.826 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:11.095+0000 7febb3a3bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-09-08T17:15:12.826 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2020-09-08T17:15:12.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2020-09-08T17:15:12.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2020-09-08T17:15:12.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2020-09-08T17:15:12.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2020-09-08T17:15:12.827 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:15:12.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2020-09-08T17:15:12.828 INFO:tasks.workunit.client.0.smithi120.stderr:0 2020-09-08T17:15:12.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:15:12.828 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:15:12.828 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:11.610+0000 7febb3a3bec0 -1 Falling back to public interface 2020-09-08T17:15:12.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2020-09-08T17:15:12.829 INFO:tasks.workunit.client.0.smithi120.stderr:2020-09-08T17:13:12.471+0000 7febb3a3bec0 -1 osd.2 39 log_to_monitors {default=true} 2020-09-08T17:15:12.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2020-09-08T17:15:12.829 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2020-09-08T17:15:12.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2020-09-08T17:15:12.830 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:15:12.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2020-09-08T17:15:12.830 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2020-09-08T17:15:12.830 INFO:tasks.workunit.client.0.smithi120.stderr:osd.2 up in weight 1 up_from 43 up_thru 35 down_at 40 last_clean_interval [18,39) [v2:127.0.0.1:6818/184412,v1:127.0.0.1:6819/184412] [v2:127.0.0.1:6820/184412,v1:127.0.0.1:6821/184412] exists,up 450871d5-a0a3-4f4a-ab84-960c5999c21f 2020-09-08T17:15:12.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2020-09-08T17:15:12.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2020-09-08T17:15:12.831 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2020-09-08T17:15:13.034 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2020-09-08T17:17:20.591 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2020-09-08T17:17:20.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2020-09-08T17:17:20.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2020-09-08T17:17:20.794 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-09-08T17:17:20.795 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2020-09-08T17:17:20.795 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2020-09-08T17:17:20.795 INFO:tasks.workunit.client.0.smithi120.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2020-09-08T17:17:20.795 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2020-09-08T17:17:20.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2020-09-08T17:17:20.796 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2020-09-08T17:17:20.796 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2020-09-08T17:17:20.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2020-09-08T17:17:20.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2020-09-08T17:17:20.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2020-09-08T17:17:20.797 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2020-09-08T17:17:20.797 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2020-09-08T17:17:20.798 INFO:tasks.workunit.client.0.smithi120.stderr:1 2020-09-08T17:17:20.798 INFO:tasks.workunit.client.0.smithi120.stderr:2 2020-09-08T17:17:20.798 INFO:tasks.workunit.client.0.smithi120.stderr:3' 2020-09-08T17:17:20.798 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2020-09-08T17:17:20.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:17:20.799 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-09-08T17:17:20.799 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803792 2020-09-08T17:17:21.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803792 2020-09-08T17:17:21.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792' 2020-09-08T17:17:21.002 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:17:21.003 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-09-08T17:17:21.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789959 2020-09-08T17:17:21.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789959 2020-09-08T17:17:21.003 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-158913789959' 2020-09-08T17:17:21.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:17:21.004 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-09-08T17:17:21.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=184683593731 2020-09-08T17:17:21.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 184683593731 2020-09-08T17:17:21.004 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-158913789959 2-184683593731' 2020-09-08T17:17:21.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2020-09-08T17:17:21.005 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-09-08T17:17:21.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149707 2020-09-08T17:17:21.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149707 2020-09-08T17:17:21.005 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803792 1-158913789959 2-184683593731 3-111669149707' 2020-09-08T17:17:21.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:17:21.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803792 2020-09-08T17:17:21.006 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:17:21.006 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2020-09-08T17:17:21.007 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803792 2020-09-08T17:17:21.210 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:17:21.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803792 2020-09-08T17:17:21.210 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803792' 2020-09-08T17:17:21.210 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.0 seq 25769803792 2020-09-08T17:17:21.211 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:17:21.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803789 -lt 25769803792 2020-09-08T17:17:21.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2020-09-08T17:17:21.211 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2020-09-08T17:17:21.212 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2020-09-08T17:17:21.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803792 -lt 25769803792 2020-09-08T17:17:21.212 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:17:21.212 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-158913789959 2020-09-08T17:17:21.212 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:17:21.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2020-09-08T17:17:21.213 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-158913789959 2020-09-08T17:17:21.213 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:17:21.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789959 2020-09-08T17:17:21.213 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 158913789959' 2020-09-08T17:17:21.214 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.1 seq 158913789959 2020-09-08T17:17:21.214 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2020-09-08T17:17:21.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789959 -lt 158913789959 2020-09-08T17:17:21.214 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:17:21.214 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-184683593731 2020-09-08T17:17:21.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:17:21.215 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2020-09-08T17:17:21.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-184683593731 2020-09-08T17:17:21.215 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:17:21.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=184683593731 2020-09-08T17:17:21.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2020-09-08T17:17:21.216 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.2 seq 184683593731 2020-09-08T17:17:21.216 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2020-09-08T17:17:21.216 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 184683593731 -lt 184683593731 2020-09-08T17:17:21.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2020-09-08T17:17:21.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149707 2020-09-08T17:17:21.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2020-09-08T17:17:21.217 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2020-09-08T17:17:21.217 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149707 2020-09-08T17:17:21.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2020-09-08T17:17:21.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149707 2020-09-08T17:17:21.218 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149707' 2020-09-08T17:17:21.218 INFO:tasks.workunit.client.0.smithi120.stderr:waiting osd.3 seq 111669149707 2020-09-08T17:17:21.218 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2020-09-08T17:17:21.219 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149708 -lt 111669149707 2020-09-08T17:17:21.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2020-09-08T17:17:21.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:17:21.219 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:17:21.220 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 5 == 0 2020-09-08T17:17:21.422 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:17:21.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:17:21.422 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:17:21.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:17:21.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:17:21.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:17:21.423 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:17:21.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2020-09-08T17:17:21.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:17:21.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:17:21.424 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:17:21.424 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 5 2020-09-08T17:17:21.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 3 '!=' -1 2020-09-08T17:17:21.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2020-09-08T17:17:21.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=3 2020-09-08T17:17:21.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2020-09-08T17:17:21.425 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2020-09-08T17:17:21.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2020-09-08T17:17:21.426 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2020-09-08T17:17:21.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2020-09-08T17:17:21.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2020-09-08T17:17:21.426 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-09-08T17:17:21.630 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-09-08T17:17:29.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2020-09-08T17:17:29.598 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-09-08T17:17:29.598 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=5 2020-09-08T17:17:29.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2020-09-08T17:17:29.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2020-09-08T17:17:29.599 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2020-09-08T17:17:29.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 5 = 5 2020-09-08T17:17:29.599 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2020-09-08T17:17:29.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2020-09-08T17:17:29.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-59146-1 fail 2020-09-08T17:17:29.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:103: rados_get: local dir=td/test-erasure-eio 2020-09-08T17:17:29.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local poolname=pool-jerasure 2020-09-08T17:17:29.600 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local objname=obj-remove-59146-1 2020-09-08T17:17:29.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local expect=fail 2020-09-08T17:17:29.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:111: rados_get: '[' fail = fail ']' 2020-09-08T17:17:29.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:113: rados_get: rados --pool pool-jerasure get obj-remove-59146-1 td/test-erasure-eio/COPY 2020-09-08T17:17:29.601 INFO:tasks.workunit.client.0.smithi120.stderr:error getting pool-jerasure/obj-remove-59146-1: (5) Input/output error 2020-09-08T17:17:29.601 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: return 2020-09-08T17:17:29.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2020-09-08T17:17:29.602 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:289: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2020-09-08T17:17:29.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:83: delete_erasure_coded_pool: local poolname=pool-jerasure 2020-09-08T17:17:29.806 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2020-09-08T17:17:29.806 INFO:tasks.workunit.client.0.smithi120.stderr:pool 'pool-jerasure' does not exist 2020-09-08T17:17:29.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2020-09-08T17:17:29.807 INFO:tasks.workunit.client.0.smithi120.stderr:erasure-code-profile myprofile does not exist 2020-09-08T17:17:29.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: teardown td/test-erasure-eio 2020-09-08T17:17:29.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:17:29.807 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:17:29.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:17:29.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:17:29.808 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:17:29.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:17:29.808 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:17:29.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:17:29.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:17:29.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:17:29.809 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:17:29.809 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:17:29.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:17:29.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:17:29.810 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:17:29.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:17:29.810 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:17:30.014 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:17:32.054 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:17:32.054 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:17:32.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:17:32.055 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:17:32.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:17:32.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:17:32.055 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:17:32.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:17:32.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:17:32.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:17:32.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: for func in $funcs 2020-09-08T17:17:32.056 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: setup td/test-erasure-eio 2020-09-08T17:17:32.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2020-09-08T17:17:32.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2020-09-08T17:17:32.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2020-09-08T17:17:32.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-09-08T17:17:32.057 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2020-09-08T17:17:32.058 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-09-08T17:17:32.058 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-09-08T17:17:32.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-09-08T17:17:32.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-09-08T17:17:32.058 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-09-08T17:17:32.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-09-08T17:17:32.262 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-09-08T17:17:32.262 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-09-08T17:17:32.262 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-09-08T17:17:32.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-09-08T17:17:32.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-09-08T17:17:32.263 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-09-08T17:17:32.263 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:17:32.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-09-08T17:17:32.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-09-08T17:17:32.264 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-09-08T17:17:32.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-09-08T17:17:32.264 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-09-08T17:17:32.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2020-09-08T17:17:32.265 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-09-08T17:17:32.265 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:17:32.265 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146 2020-09-08T17:17:32.265 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.59146 2020-09-08T17:17:32.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-09-08T17:17:32.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-09-08T17:17:32.266 INFO:tasks.workunit.client.0.smithi120.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2020-09-08T17:17:32.266 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-09-08T17:17:32.266 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-09-08T17:17:32.470 INFO:tasks.workunit.client.0.smithi120.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.59146